postgresql-best-practices

PostgreSQL Advanced Best Practices (PostgreSQL 18+) Architecture at a Glance Skill Contents 🚀 Getting Started (Read These First) | Document | Purpose | |----------|---------| | quick-reference.md | QUICK LOOKUP - Single-page cheat sheet (print this!) | | schema-architecture.md | START HERE - Schema separation pattern (data/private/api) | | coding-standards-trivadis.md | Coding standards & naming conventions (l , g , co ) | 📚 Core Reference (Use Daily) | Document | Purpose | |----------|---------| | plpgsql-table-api.md | Table API functions, procedures, triggers | | schema-naming.md | Namin…

; -- Basic email validation\n\nDROP TABLE staging_customers;\n```\n\n### COPY in API Functions\n\n```sql\n-- Procedure to import data from application\nCREATE OR REPLACE PROCEDURE api.bulk_import_customers(\n in_csv_data text -- CSV content as text\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_imported integer;\nBEGIN\n -- Create temp table\n CREATE TEMP TABLE temp_import (\n email text,\n name text\n ) ON COMMIT DROP;\n \n -- Parse CSV data (simple implementation)\n INSERT INTO temp_import (email, name)\n SELECT \n split_part(line, ',', 1),\n split_part(line, ',', 2)\n FROM unnest(string_to_array(in_csv_data, E'\\n')) AS line\n WHERE line != '' AND line NOT LIKE 'email%'; -- Skip header\n \n -- Insert with deduplication\n INSERT INTO data.customers (email, name)\n SELECT DISTINCT lower(trim(email)), trim(name)\n FROM temp_import t\n WHERE NOT EXISTS (\n SELECT 1 FROM data.customers c \n WHERE c.email = lower(trim(t.email))\n );\n \n GET DIAGNOSTICS l_imported = ROW_COUNT;\n RAISE NOTICE 'Imported % customers', l_imported;\nEND;\n$;\n```\n\n## Batch INSERT Patterns\n\n### Multi-Row INSERT\n\n```sql\n-- Multiple rows in single statement (up to ~1000 rows per statement)\nINSERT INTO data.order_items (order_id, product_id, quantity, unit_price)\nVALUES \n ('order-1', 'product-a', 2, 10.00),\n ('order-1', 'product-b', 1, 25.00),\n ('order-1', 'product-c', 3, 15.00);\n\n-- Returns all inserted IDs\nINSERT INTO data.customers (email, name)\nVALUES \n ('[email protected]', 'Alice'),\n ('[email protected]', 'Bob'),\n ('[email protected]', 'Charlie')\nRETURNING id, email;\n```\n\n### INSERT with Array Parameters\n\n```sql\n-- API procedure accepting arrays\nCREATE OR REPLACE PROCEDURE api.bulk_insert_order_items(\n in_order_id uuid,\n in_product_ids uuid[],\n in_quantities integer[],\n in_prices numeric[]\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_inserted integer;\nBEGIN\n -- Validate arrays have same length\n IF array_length(in_product_ids, 1) != array_length(in_quantities, 1)\n OR array_length(in_product_ids, 1) != array_length(in_prices, 1) THEN\n RAISE EXCEPTION 'Array lengths must match';\n END IF;\n \n INSERT INTO data.order_items (order_id, product_id, quantity, unit_price)\n SELECT \n in_order_id,\n unnest(in_product_ids),\n unnest(in_quantities),\n unnest(in_prices);\n \n GET DIAGNOSTICS l_inserted = ROW_COUNT;\n RAISE NOTICE 'Inserted % order items', l_inserted;\nEND;\n$;\n\n-- Usage\nCALL api.bulk_insert_order_items(\n 'order-uuid',\n ARRAY['prod-1', 'prod-2', 'prod-3']::uuid[],\n ARRAY[2, 1, 3],\n ARRAY[10.00, 25.00, 15.00]\n);\n```\n\n### INSERT from JSONB Array\n\n```sql\n-- API procedure accepting JSONB array\nCREATE OR REPLACE PROCEDURE api.bulk_insert_from_json(\n in_items jsonb -- [{\"email\": \"[email protected]\", \"name\": \"Alice\"}, ...]\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_inserted integer;\nBEGIN\n INSERT INTO data.customers (email, name)\n SELECT \n item->>'email',\n item->>'name'\n FROM jsonb_array_elements(in_items) AS item\n WHERE NOT EXISTS (\n SELECT 1 FROM data.customers c \n WHERE c.email = item->>'email'\n );\n \n GET DIAGNOSTICS l_inserted = ROW_COUNT;\n RAISE NOTICE 'Inserted % records', l_inserted;\nEND;\n$;\n\n-- Usage\nCALL api.bulk_insert_from_json('[\n {\"email\": \"[email protected]\", \"name\": \"Alice\"},\n {\"email\": \"[email protected]\", \"name\": \"Bob\"}\n]'::jsonb);\n```\n\n### INSERT with SELECT\n\n```sql\n-- Copy data between tables\nINSERT INTO data.order_archive (id, customer_id, total, status, created_at)\nSELECT id, customer_id, total, status, created_at\nFROM data.orders\nWHERE status = 'completed'\n AND created_at \u003c now() - interval '1 year';\n\n-- Insert with transformation\nINSERT INTO data.monthly_summary (month, total_orders, total_revenue)\nSELECT \n date_trunc('month', created_at)::date,\n COUNT(*),\n SUM(total)\nFROM data.orders\nWHERE created_at >= '2024-01-01'\nGROUP BY date_trunc('month', created_at);\n```\n\n## UPSERT Patterns\n\n### Basic ON CONFLICT\n\n```sql\n-- Update on duplicate key\nINSERT INTO data.customers (email, name)\nVALUES ('[email protected]', 'John Doe')\nON CONFLICT (email) DO UPDATE SET\n name = EXCLUDED.name,\n updated_at = now();\n\n-- Insert or ignore\nINSERT INTO data.tags (name)\nVALUES ('sale'), ('new'), ('featured')\nON CONFLICT (name) DO NOTHING;\n```\n\n### Conditional UPDATE\n\n```sql\n-- Only update if data actually changed\nINSERT INTO data.products (sku, name, price)\nVALUES ('SKU-001', 'Widget', 29.99)\nON CONFLICT (sku) DO UPDATE SET\n name = EXCLUDED.name,\n price = EXCLUDED.price,\n updated_at = now()\nWHERE \n data.products.name IS DISTINCT FROM EXCLUDED.name\n OR data.products.price IS DISTINCT FROM EXCLUDED.price;\n```\n\n### Bulk UPSERT\n\n```sql\n-- Upsert with arrays\nCREATE OR REPLACE PROCEDURE api.bulk_upsert_products(\n in_skus text[],\n in_names text[],\n in_prices numeric[]\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_upserted integer;\nBEGIN\n INSERT INTO data.products (sku, name, price)\n SELECT unnest(in_skus), unnest(in_names), unnest(in_prices)\n ON CONFLICT (sku) DO UPDATE SET\n name = EXCLUDED.name,\n price = EXCLUDED.price,\n updated_at = now()\n WHERE data.products.name IS DISTINCT FROM EXCLUDED.name\n OR data.products.price IS DISTINCT FROM EXCLUDED.price;\n \n GET DIAGNOSTICS l_upserted = ROW_COUNT;\n RAISE NOTICE 'Upserted % products', l_upserted;\nEND;\n$;\n```\n\n### Upsert with RETURNING\n\n```sql\n-- Get info about what was inserted vs updated\nWITH upsert AS (\n INSERT INTO data.customers (email, name)\n VALUES ('[email protected]', 'John Doe')\n ON CONFLICT (email) DO UPDATE SET\n name = EXCLUDED.name,\n updated_at = now()\n RETURNING id, email, (xmax = 0) AS inserted\n)\nSELECT \n id, \n email,\n CASE WHEN inserted THEN 'created' ELSE 'updated' END AS action\nFROM upsert;\n```\n\n## Batch UPDATE Patterns\n\n### UPDATE with JOIN\n\n```sql\n-- Update from another table\nUPDATE data.products p\nSET price = np.new_price,\n updated_at = now()\nFROM data.new_prices np\nWHERE p.sku = np.sku\n AND p.price IS DISTINCT FROM np.new_price;\n\n-- Update with aggregated data\nUPDATE data.customers c\nSET total_orders = sub.order_count,\n total_spent = sub.total_amount\nFROM (\n SELECT \n customer_id,\n COUNT(*) AS order_count,\n SUM(total) AS total_amount\n FROM data.orders\n WHERE status = 'completed'\n GROUP BY customer_id\n) sub\nWHERE c.id = sub.customer_id;\n```\n\n### UPDATE with Arrays\n\n```sql\n-- Batch update using arrays\nCREATE OR REPLACE PROCEDURE api.bulk_update_status(\n in_ids uuid[],\n in_status text\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_updated integer;\nBEGIN\n UPDATE data.orders\n SET status = in_status,\n updated_at = now()\n WHERE id = ANY(in_ids);\n \n GET DIAGNOSTICS l_updated = ROW_COUNT;\n RAISE NOTICE 'Updated % orders', l_updated;\nEND;\n$;\n\n-- Update with different values per row\nCREATE OR REPLACE PROCEDURE api.bulk_update_prices(\n in_product_ids uuid[],\n in_prices numeric[]\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n UPDATE data.products p\n SET price = updates.new_price,\n updated_at = now()\n FROM (\n SELECT \n unnest(in_product_ids) AS id,\n unnest(in_prices) AS new_price\n ) updates\n WHERE p.id = updates.id\n AND p.price IS DISTINCT FROM updates.new_price;\nEND;\n$;\n```\n\n### UPDATE with LIMIT (Chunked)\n\n```sql\n-- Update in batches to avoid long locks\nCREATE OR REPLACE PROCEDURE api.migrate_data_chunked(\n in_batch_size integer DEFAULT 1000\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_updated integer;\n l_total integer := 0;\nBEGIN\n LOOP\n WITH batch AS (\n SELECT id\n FROM data.legacy_table\n WHERE migrated = false\n LIMIT in_batch_size\n FOR UPDATE SKIP LOCKED\n )\n UPDATE data.legacy_table t\n SET migrated = true,\n new_column = compute_new_value(t.old_column)\n FROM batch\n WHERE t.id = batch.id;\n \n GET DIAGNOSTICS l_updated = ROW_COUNT;\n l_total := l_total + l_updated;\n \n EXIT WHEN l_updated = 0;\n \n COMMIT; -- Release locks between batches\n RAISE NOTICE 'Processed % records (total: %)', l_updated, l_total;\n END LOOP;\n \n RAISE NOTICE 'Migration complete. Total: % records', l_total;\nEND;\n$;\n```\n\n## Batch DELETE Patterns\n\n### DELETE with LIMIT\n\n```sql\n-- Delete in batches\nCREATE OR REPLACE PROCEDURE api.purge_old_logs(\n in_older_than interval DEFAULT interval '90 days',\n in_batch_size integer DEFAULT 10000\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_deleted integer;\n l_total integer := 0;\n l_cutoff timestamptz;\nBEGIN\n l_cutoff := now() - in_older_than;\n \n LOOP\n DELETE FROM data.logs\n WHERE id IN (\n SELECT id FROM data.logs\n WHERE created_at \u003c l_cutoff\n LIMIT in_batch_size\n );\n \n GET DIAGNOSTICS l_deleted = ROW_COUNT;\n l_total := l_total + l_deleted;\n \n EXIT WHEN l_deleted = 0;\n \n COMMIT;\n RAISE NOTICE 'Deleted % records (total: %)', l_deleted, l_total;\n \n -- Optional: Add delay to reduce system load\n PERFORM pg_sleep(0.1);\n END LOOP;\n \n RAISE NOTICE 'Purge complete. Total deleted: %', l_total;\nEND;\n$;\n```\n\n### DELETE with Archive\n\n```sql\n-- Move to archive before deleting\nCREATE OR REPLACE PROCEDURE api.archive_and_delete_orders(\n in_older_than interval DEFAULT interval '1 year'\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_archived integer;\nBEGIN\n -- Archive\n INSERT INTO data.orders_archive\n SELECT * FROM data.orders\n WHERE created_at \u003c now() - in_older_than\n AND status IN ('completed', 'cancelled');\n \n GET DIAGNOSTICS l_archived = ROW_COUNT;\n \n -- Delete archived records\n DELETE FROM data.orders\n WHERE created_at \u003c now() - in_older_than\n AND status IN ('completed', 'cancelled');\n \n RAISE NOTICE 'Archived and deleted % orders', l_archived;\nEND;\n$;\n```\n\n### TRUNCATE (Fastest Delete All)\n\n```sql\n-- Much faster than DELETE for removing all rows\nTRUNCATE data.temp_import;\n\n-- Truncate with restart identity\nTRUNCATE data.logs RESTART IDENTITY;\n\n-- Truncate cascade (also truncates dependent tables)\nTRUNCATE data.customers CASCADE;\n\n-- Note: TRUNCATE requires table lock, cannot be rolled back in some cases\n```\n\n## Processing Large Result Sets\n\n### Server-Side Cursor\n\n```sql\n-- Declare cursor for large result set\nCREATE OR REPLACE PROCEDURE api.process_large_dataset()\nLANGUAGE plpgsql\nAS $\nDECLARE\n c_orders CURSOR FOR \n SELECT id, customer_id, total \n FROM data.orders \n WHERE status = 'pending';\n l_batch_size integer := 100;\n l_records RECORD[];\n l_count integer := 0;\nBEGIN\n FOR l_record IN c_orders LOOP\n -- Process each record\n PERFORM private.process_order(l_record.id);\n l_count := l_count + 1;\n \n -- Periodic commit\n IF l_count % l_batch_size = 0 THEN\n COMMIT;\n RAISE NOTICE 'Processed % records', l_count;\n END IF;\n END LOOP;\n \n RAISE NOTICE 'Total processed: %', l_count;\nEND;\n$;\n```\n\n### FETCH with LIMIT\n\n```sql\n-- Paginated processing\nCREATE OR REPLACE PROCEDURE api.process_in_pages(\n in_page_size integer DEFAULT 1000\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_last_id uuid := '00000000-0000-0000-0000-000000000000';\n l_count integer;\nBEGIN\n LOOP\n -- Process one page\n WITH page AS (\n SELECT id, customer_id, total\n FROM data.orders\n WHERE id > l_last_id\n ORDER BY id\n LIMIT in_page_size\n )\n UPDATE data.orders o\n SET processed = true\n FROM page p\n WHERE o.id = p.id\n RETURNING o.id INTO l_last_id;\n \n GET DIAGNOSTICS l_count = ROW_COUNT;\n \n EXIT WHEN l_count = 0;\n \n COMMIT;\n RAISE NOTICE 'Processed page, last_id: %', l_last_id;\n END LOOP;\nEND;\n$;\n```\n\n## Temporary Tables for Staging\n\n### Staging Pattern\n\n```sql\nCREATE OR REPLACE PROCEDURE api.import_with_validation(\n in_data jsonb\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_valid integer;\n l_invalid integer;\nBEGIN\n -- Create staging table\n CREATE TEMP TABLE staging (\n email text,\n name text,\n is_valid boolean DEFAULT true,\n error_message text\n ) ON COMMIT DROP;\n \n -- Load data\n INSERT INTO staging (email, name)\n SELECT \n item->>'email',\n item->>'name'\n FROM jsonb_array_elements(in_data) AS item;\n \n -- Validate: email format\n UPDATE staging\n SET is_valid = false,\n error_message = 'Invalid email format'\n WHERE email !~ '^[^@]+@[^@]+\\.[^@]+

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

;\n \n -- Validate: duplicate emails\n UPDATE staging s\n SET is_valid = false,\n error_message = 'Duplicate email'\n WHERE EXISTS (\n SELECT 1 FROM data.customers c WHERE c.email = s.email\n );\n \n -- Count results\n SELECT COUNT(*) FILTER (WHERE is_valid) INTO l_valid FROM staging;\n SELECT COUNT(*) FILTER (WHERE NOT is_valid) INTO l_invalid FROM staging;\n \n -- Insert valid records\n INSERT INTO data.customers (email, name)\n SELECT email, name FROM staging WHERE is_valid;\n \n -- Log invalid records\n INSERT INTO data.import_errors (email, name, error_message, imported_at)\n SELECT email, name, error_message, now()\n FROM staging WHERE NOT is_valid;\n \n RAISE NOTICE 'Imported: %, Rejected: %', l_valid, l_invalid;\nEND;\n$;\n```\n\n### Unlogged Tables for Speed\n\n```sql\n-- Unlogged tables are faster but not crash-safe\nCREATE UNLOGGED TABLE data.temp_calculations (\n id uuid PRIMARY KEY,\n result numeric,\n processed boolean DEFAULT false\n);\n\n-- Use for intermediate results that can be regenerated\n-- Don't use for permanent data!\n```\n\n## Transaction Management\n\n### Autonomous Operations (Logging)\n\n```sql\n-- PostgreSQL doesn't have autonomous transactions\n-- Use dblink for separate transaction or log to external system\n\n-- Alternative: Use SAVEPOINT for partial rollback\nCREATE OR REPLACE PROCEDURE api.process_with_logging()\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_record RECORD;\nBEGIN\n FOR l_record IN SELECT * FROM data.pending_items LOOP\n SAVEPOINT item_savepoint;\n \n BEGIN\n -- Process item\n PERFORM private.process_item(l_record.id);\n EXCEPTION\n WHEN OTHERS THEN\n -- Rollback just this item\n ROLLBACK TO SAVEPOINT item_savepoint;\n \n -- Log error (still in same transaction)\n INSERT INTO data.processing_errors (item_id, error)\n VALUES (l_record.id, SQLERRM);\n END;\n \n RELEASE SAVEPOINT item_savepoint;\n END LOOP;\nEND;\n$;\n```\n\n### Batch Commits\n\n```sql\n-- Commit periodically during long operations\nCREATE OR REPLACE PROCEDURE api.long_running_process()\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_count integer := 0;\n co_batch_size CONSTANT integer := 1000;\nBEGIN\n FOR l_record IN SELECT * FROM data.items_to_process LOOP\n -- Do work\n UPDATE data.items_to_process \n SET processed = true \n WHERE id = l_record.id;\n \n l_count := l_count + 1;\n \n -- Commit every batch\n IF l_count % co_batch_size = 0 THEN\n COMMIT;\n END IF;\n END LOOP;\nEND;\n$;\n```\n\n### Advisory Locks for Coordination\n\n```sql\n-- Ensure only one instance runs\nCREATE OR REPLACE PROCEDURE api.exclusive_batch_job()\nLANGUAGE plpgsql\nAS $\nDECLARE\n co_lock_id CONSTANT bigint := 12345;\n l_acquired boolean;\nBEGIN\n -- Try to acquire lock\n SELECT pg_try_advisory_lock(co_lock_id) INTO l_acquired;\n \n IF NOT l_acquired THEN\n RAISE NOTICE 'Another instance is running, exiting';\n RETURN;\n END IF;\n \n -- Do batch work\n BEGIN\n -- ... your batch logic here ...\n RAISE NOTICE 'Batch job completed';\n EXCEPTION\n WHEN OTHERS THEN\n -- Always release lock\n PERFORM pg_advisory_unlock(co_lock_id);\n RAISE;\n END;\n \n -- Release lock\n PERFORM pg_advisory_unlock(co_lock_id);\nEND;\n$;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":18465,"content_sha256":"d67d51889b65b39f38b276c0d06362ddd4dd57d07df4322bba69a9ec915b761c"},{"filename":"references/checklists-troubleshooting.md","content":"# Checklists & Troubleshooting\n\nQuick reference checklists for common tasks and solutions to common problems.\n\n> **Related**: See [quick-reference.md](quick-reference.md) for patterns and [anti-patterns.md](anti-patterns.md) for what to avoid.\n\n---\n\n## Checklist: New Project Setup\n\n- [ ] Create schemas: `data`, `private`, `api`, `app_audit`\n- [ ] Revoke public schema access: `REVOKE ALL ON SCHEMA public FROM PUBLIC`\n- [ ] Install migration system (run `scripts/001_install_migration_system.sql`)\n- [ ] Create application roles with appropriate permissions\n- [ ] Set up `private.set_updated_at()` trigger function\n\n## Checklist: New Table\n\n- [ ] Create table in `data` schema\n- [ ] Use `uuidv7()` or `GENERATED ALWAYS AS IDENTITY` for primary key\n- [ ] Add `created_at timestamptz NOT NULL DEFAULT now()`\n- [ ] Add `updated_at timestamptz NOT NULL DEFAULT now()`\n- [ ] Apply `private.set_updated_at()` trigger\n- [ ] Create indexes for foreign keys\n- [ ] Create indexes for common query patterns\n- [ ] Create API functions in `api` schema\n\n## Checklist: API Function\n\n- [ ] Place in `api` schema\n- [ ] Add `SECURITY DEFINER`\n- [ ] Add `SET search_path = data, private, pg_temp`\n- [ ] Use explicit `RETURNS TABLE (...)` (never `RETURNS SETOF table`)\n- [ ] Prefix parameters with `in_`, outputs with `io_`\n- [ ] Add appropriate volatility (`STABLE` for reads, default for writes)\n- [ ] Add comments\n\n## Checklist: Security Review\n\n- [ ] No direct grants on `data` or `private` schemas\n- [ ] All `api` functions use `SECURITY DEFINER` with `SET search_path`\n- [ ] Sensitive columns (passwords, tokens) never returned by API functions\n- [ ] Application role has only `EXECUTE` on `api` schema\n\n---\n\n## Troubleshooting\n\n### \"Permission denied for table...\"\n\n**Cause**: Application trying to access `data` schema directly.\n\n**Fix**: Access data through `api` functions only. Check that:\n1. Function uses `SECURITY DEFINER`\n2. Function has `SET search_path = data, private, pg_temp`\n3. Application role has `EXECUTE` permission on the function\n\n### \"Migration lock not available\"\n\n**Cause**: Another migration is running or crashed without releasing lock.\n\n**Fix**:\n```sql\n-- Check who holds the lock\nSELECT * FROM app_migration.get_lock_holder();\n\n-- If the session is gone, the lock will auto-release\n-- If stuck, check for orphaned advisory locks:\nSELECT * FROM pg_locks WHERE locktype = 'advisory';\n```\n\n### \"Checksum mismatch for version...\"\n\n**Cause**: A versioned migration was modified after execution.\n\n**Fix**: Versioned migrations should never be modified. Either:\n1. Create a new migration to make changes\n2. If in development, clear and re-run: `CALL app_migration.clear_failed();`\n\n### Function returns wrong columns\n\n**Cause**: Using `RETURNS SETOF table` exposes all columns.\n\n**Fix**: Use explicit `RETURNS TABLE (col1 type, col2 type, ...)` to control output.\n\n### Slow queries\n\n**Check**:\n1. Is there an index on the WHERE clause columns?\n2. Is the query using the index? (`EXPLAIN ANALYZE`)\n3. For foreign keys, is there an index on the FK column?\n\nSee [performance-tuning.md](performance-tuning.md) for detailed optimization strategies.\n\n### SECURITY DEFINER function not working\n\n**Cause**: Missing `SET search_path` allows search path manipulation attacks.\n\n**Fix**: Always pair `SECURITY DEFINER` with `SET search_path`:\n```sql\nCREATE FUNCTION api.my_function(...)\nRETURNS ...\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp -- Required!\nAS $\n...\n$;\n```\n\n### Trigger not firing\n\n**Cause**: Trigger may be disabled or on wrong timing.\n\n**Check**:\n```sql\n-- List triggers on table\nSELECT tgname, tgenabled, tgtype \nFROM pg_trigger \nWHERE tgrelid = 'data.my_table'::regclass;\n\n-- Enable if disabled\nALTER TABLE data.my_table ENABLE TRIGGER my_trigger;\n```\n\n### UUID vs IDENTITY confusion\n\n**Use UUIDv7 when**:\n- Distributed systems (no coordination needed)\n- URLs/external references (no sequence guessing)\n- Time-ordered sorting needed\n\n**Use IDENTITY when**:\n- Internal IDs only\n- Need compact storage (8 bytes vs 16)\n- Sequential inserts matter for performance\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":4110,"content_sha256":"87c94199aa6c5862b7363b06d76351df4b631fb372435f714abf8160e1a035b1"},{"filename":"references/cicd-integration.md","content":"# CI/CD Integration for PostgreSQL\n\nThis document provides templates and patterns for integrating PostgreSQL database changes into CI/CD pipelines.\n\n## Table of Contents\n\n1. [GitHub Actions](#github-actions)\n2. [GitLab CI](#gitlab-ci)\n3. [Docker Setup](#docker-setup)\n4. [Database Testing Pipeline](#database-testing-pipeline)\n5. [Migration Validation](#migration-validation)\n6. [Schema Comparison](#schema-comparison)\n7. [Deployment Strategies](#deployment-strategies)\n\n## GitHub Actions\n\n### Basic Migration Workflow\n\n```yaml\n# .github/workflows/database.yml\nname: Database CI\n\non:\n push:\n branches: [main, develop]\n paths:\n - 'db/**'\n pull_request:\n branches: [main]\n paths:\n - 'db/**'\n\nenv:\n POSTGRES_USER: test\n POSTGRES_PASSWORD: test\n POSTGRES_DB: test_db\n\njobs:\n test-migrations:\n runs-on: ubuntu-latest\n \n services:\n postgres:\n image: postgres:18\n env:\n POSTGRES_USER: ${{ env.POSTGRES_USER }}\n POSTGRES_PASSWORD: ${{ env.POSTGRES_PASSWORD }}\n POSTGRES_DB: ${{ env.POSTGRES_DB }}\n ports:\n - 5432:5432\n options: >-\n --health-cmd pg_isready\n --health-interval 10s\n --health-timeout 5s\n --health-retries 5\n\n steps:\n - uses: actions/checkout@v4\n \n - name: Install PostgreSQL client\n run: |\n sudo apt-get update\n sudo apt-get install -y postgresql-client\n \n - name: Wait for PostgreSQL\n run: |\n until pg_isready -h localhost -p 5432 -U $POSTGRES_USER; do\n echo \"Waiting for postgres...\"\n sleep 2\n done\n \n - name: Run migrations\n env:\n PGHOST: localhost\n PGUSER: ${{ env.POSTGRES_USER }}\n PGPASSWORD: ${{ env.POSTGRES_PASSWORD }}\n PGDATABASE: ${{ env.POSTGRES_DB }}\n run: |\n # Install migration system\n psql -f db/scripts/001_install_migration_system.sql\n psql -f db/scripts/002_migration_runner_helpers.sql\n \n # Run all migrations\n for f in db/migrations/V*.sql; do\n echo \"Running: $f\"\n psql -f \"$f\"\n done\n \n # Run repeatable migrations\n for f in db/migrations/R__*.sql; do\n echo \"Running: $f\"\n psql -f \"$f\"\n done\n \n - name: Run tests\n env:\n PGHOST: localhost\n PGUSER: ${{ env.POSTGRES_USER }}\n PGPASSWORD: ${{ env.POSTGRES_PASSWORD }}\n PGDATABASE: ${{ env.POSTGRES_DB }}\n run: |\n psql -f db/tests/install_tests.sql\n psql -c \"SELECT * FROM test.run_all_tests();\" | tee test_results.txt\n \n - name: Check test results\n run: |\n if grep -q \"FAIL\\|ERROR\" test_results.txt; then\n echo \"::error::Database tests failed\"\n exit 1\n fi\n echo \"All tests passed!\"\n \n - name: Upload test results\n uses: actions/upload-artifact@v3\n if: always()\n with:\n name: test-results\n path: test_results.txt\n```\n\n### Migration Validation Workflow\n\n```yaml\n# .github/workflows/validate-migrations.yml\nname: Validate Migrations\n\non:\n pull_request:\n paths:\n - 'db/migrations/**'\n\njobs:\n validate:\n runs-on: ubuntu-latest\n \n services:\n postgres:\n image: postgres:18\n env:\n POSTGRES_USER: test\n POSTGRES_PASSWORD: test\n POSTGRES_DB: test_db\n ports:\n - 5432:5432\n options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5\n\n steps:\n - uses: actions/checkout@v4\n with:\n fetch-depth: 0 # Need full history for diff\n \n - name: Get changed migrations\n id: changed\n run: |\n CHANGED=$(git diff --name-only origin/${{ github.base_ref }}...HEAD -- 'db/migrations/*.sql')\n echo \"files=$CHANGED\" >> $GITHUB_OUTPUT\n echo \"Changed files: $CHANGED\"\n \n - name: Validate migration naming\n run: |\n for f in ${{ steps.changed.outputs.files }}; do\n filename=$(basename \"$f\")\n # Check versioned migrations\n if [[ $filename == V* ]]; then\n if ! [[ $filename =~ ^V[0-9]{3}__[a-z_]+\\.sql$ ]]; then\n echo \"::error::Invalid migration name: $filename\"\n echo \"Expected format: V001__description_here.sql\"\n exit 1\n fi\n fi\n # Check repeatable migrations \n if [[ $filename == R__* ]]; then\n if ! [[ $filename =~ ^R__[a-z_]+\\.sql$ ]]; then\n echo \"::error::Invalid repeatable migration name: $filename\"\n echo \"Expected format: R__description.sql\"\n exit 1\n fi\n fi\n done\n echo \"Migration naming validation passed\"\n \n - name: Check for destructive operations\n run: |\n WARNINGS=\"\"\n for f in ${{ steps.changed.outputs.files }}; do\n if grep -qiE \"DROP\\s+TABLE|TRUNCATE|DELETE\\s+FROM.*WHERE\\s+1\\s*=\\s*1\" \"$f\"; then\n WARNINGS=\"$WARNINGS\\n⚠️ $f contains potentially destructive operations\"\n fi\n done\n if [ -n \"$WARNINGS\" ]; then\n echo -e \"::warning::$WARNINGS\"\n fi\n \n - name: Syntax check\n env:\n PGHOST: localhost\n PGUSER: test\n PGPASSWORD: test\n PGDATABASE: test_db\n run: |\n for f in ${{ steps.changed.outputs.files }}; do\n echo \"Checking syntax: $f\"\n # Use EXPLAIN to check syntax without executing\n psql -c \"BEGIN; \\i $f ROLLBACK;\" 2>&1 || {\n echo \"::error::Syntax error in $f\"\n exit 1\n }\n done\n```\n\n### Deployment Workflow\n\n```yaml\n# .github/workflows/deploy.yml\nname: Deploy Database\n\non:\n push:\n branches: [main]\n paths:\n - 'db/**'\n\njobs:\n deploy-staging:\n runs-on: ubuntu-latest\n environment: staging\n \n steps:\n - uses: actions/checkout@v4\n \n - name: Deploy to staging\n env:\n PGHOST: ${{ secrets.STAGING_DB_HOST }}\n PGUSER: ${{ secrets.STAGING_DB_USER }}\n PGPASSWORD: ${{ secrets.STAGING_DB_PASSWORD }}\n PGDATABASE: ${{ secrets.STAGING_DB_NAME }}\n run: |\n ./scripts/deploy-migrations.sh\n \n - name: Run smoke tests\n env:\n PGHOST: ${{ secrets.STAGING_DB_HOST }}\n PGUSER: ${{ secrets.STAGING_DB_USER }}\n PGPASSWORD: ${{ secrets.STAGING_DB_PASSWORD }}\n PGDATABASE: ${{ secrets.STAGING_DB_NAME }}\n run: |\n psql -c \"SELECT api.healthcheck();\"\n\n deploy-production:\n needs: deploy-staging\n runs-on: ubuntu-latest\n environment: production\n \n steps:\n - uses: actions/checkout@v4\n \n - name: Create backup\n env:\n PGHOST: ${{ secrets.PROD_DB_HOST }}\n PGUSER: ${{ secrets.PROD_DB_USER }}\n PGPASSWORD: ${{ secrets.PROD_DB_PASSWORD }}\n PGDATABASE: ${{ secrets.PROD_DB_NAME }}\n run: |\n pg_dump -Fc > backup_$(date +%Y%m%d_%H%M%S).dump\n # Upload to S3 or other backup storage\n \n - name: Deploy to production\n env:\n PGHOST: ${{ secrets.PROD_DB_HOST }}\n PGUSER: ${{ secrets.PROD_DB_USER }}\n PGPASSWORD: ${{ secrets.PROD_DB_PASSWORD }}\n PGDATABASE: ${{ secrets.PROD_DB_NAME }}\n run: |\n ./scripts/deploy-migrations.sh\n```\n\n## GitLab CI\n\n```yaml\n# .gitlab-ci.yml\nstages:\n - test\n - validate\n - deploy\n\nvariables:\n POSTGRES_USER: test\n POSTGRES_PASSWORD: test\n POSTGRES_DB: test_db\n\n.db-test-template:\n image: postgres:18\n services:\n - postgres:18\n variables:\n PGHOST: postgres\n PGUSER: $POSTGRES_USER\n PGPASSWORD: $POSTGRES_PASSWORD\n PGDATABASE: $POSTGRES_DB\n before_script:\n - apt-get update && apt-get install -y postgresql-client\n - until pg_isready -h $PGHOST; do sleep 1; done\n\ntest-migrations:\n extends: .db-test-template\n stage: test\n script:\n - psql -f db/scripts/001_install_migration_system.sql\n - psql -f db/scripts/002_migration_runner_helpers.sql\n - for f in db/migrations/V*.sql; do psql -f \"$f\"; done\n - psql -f db/tests/install_tests.sql\n - psql -c \"SELECT * FROM test.run_all_tests();\" | tee test_results.txt\n - \"! grep -q 'FAIL\\\\|ERROR' test_results.txt\"\n artifacts:\n paths:\n - test_results.txt\n when: always\n only:\n changes:\n - db/**/*\n\nvalidate-schema:\n extends: .db-test-template\n stage: validate\n script:\n - ./scripts/validate-schema.sh\n only:\n refs:\n - merge_requests\n changes:\n - db/**/*\n\ndeploy-staging:\n stage: deploy\n environment:\n name: staging\n script:\n - ./scripts/deploy-migrations.sh\n only:\n - main\n when: manual\n\ndeploy-production:\n stage: deploy\n environment:\n name: production\n script:\n - ./scripts/deploy-migrations.sh\n only:\n - main\n when: manual\n needs:\n - deploy-staging\n```\n\n## Docker Setup\n\n### Dockerfile for Testing\n\n```dockerfile\n# Dockerfile.db-test\nFROM postgres:18\n\n# Install pgTAP for testing\nRUN apt-get update && apt-get install -y \\\n postgresql-18-pgtap \\\n make \\\n && rm -rf /var/lib/apt/lists/*\n\n# Copy initialization scripts\nCOPY db/scripts/*.sql /docker-entrypoint-initdb.d/00-scripts/\nCOPY db/migrations/V*.sql /docker-entrypoint-initdb.d/01-versioned/\nCOPY db/migrations/R__*.sql /docker-entrypoint-initdb.d/02-repeatable/\nCOPY db/tests/*.sql /docker-entrypoint-initdb.d/03-tests/\n\n# Copy test runner\nCOPY scripts/run-db-tests.sh /docker-entrypoint-initdb.d/99-run-tests.sh\nRUN chmod +x /docker-entrypoint-initdb.d/99-run-tests.sh\n```\n\n### Docker Compose for Development\n\n```yaml\n# docker-compose.yml\nversion: '3.8'\n\nservices:\n db:\n image: postgres:18\n environment:\n POSTGRES_USER: dev\n POSTGRES_PASSWORD: dev\n POSTGRES_DB: myapp_dev\n ports:\n - \"5432:5432\"\n volumes:\n - postgres_data:/var/lib/postgresql/data\n - ./db/scripts:/docker-entrypoint-initdb.d/scripts:ro\n healthcheck:\n test: [\"CMD-SHELL\", \"pg_isready -U dev -d myapp_dev\"]\n interval: 5s\n timeout: 5s\n retries: 5\n\n db-test:\n build:\n context: .\n dockerfile: Dockerfile.db-test\n environment:\n POSTGRES_USER: test\n POSTGRES_PASSWORD: test\n POSTGRES_DB: myapp_test\n ports:\n - \"5433:5432\"\n\n migrate:\n image: postgres:18\n depends_on:\n db:\n condition: service_healthy\n environment:\n PGHOST: db\n PGUSER: dev\n PGPASSWORD: dev\n PGDATABASE: myapp_dev\n volumes:\n - ./db:/db:ro\n - ./scripts:/scripts:ro\n command: [\"/scripts/run-migrations.sh\"]\n\nvolumes:\n postgres_data:\n```\n\n### Test Docker Compose\n\n```yaml\n# docker-compose.test.yml\nversion: '3.8'\n\nservices:\n db:\n image: postgres:18\n environment:\n POSTGRES_USER: test\n POSTGRES_PASSWORD: test\n POSTGRES_DB: test_db\n tmpfs:\n - /var/lib/postgresql/data # Use tmpfs for speed\n healthcheck:\n test: [\"CMD-SHELL\", \"pg_isready\"]\n interval: 2s\n timeout: 2s\n retries: 10\n\n test-runner:\n image: postgres:18\n depends_on:\n db:\n condition: service_healthy\n environment:\n PGHOST: db\n PGUSER: test\n PGPASSWORD: test\n PGDATABASE: test_db\n volumes:\n - ./db:/db:ro\n - ./scripts:/scripts:ro\n command: [\"/scripts/run-all-tests.sh\"]\n```\n\n## Database Testing Pipeline\n\n### Test Runner Script\n\n```bash\n#!/bin/bash\n# scripts/run-all-tests.sh\n\nset -e\n\necho \"=== Installing migration system ===\"\npsql -f /db/scripts/001_install_migration_system.sql\npsql -f /db/scripts/002_migration_runner_helpers.sql\n\necho \"=== Running versioned migrations ===\"\nfor f in /db/migrations/V*.sql; do\n if [ -f \"$f\" ]; then\n echo \"Running: $f\"\n psql -f \"$f\"\n fi\ndone\n\necho \"=== Running repeatable migrations ===\"\nfor f in /db/migrations/R__*.sql; do\n if [ -f \"$f\" ]; then\n echo \"Running: $f\"\n psql -f \"$f\"\n fi\ndone\n\necho \"=== Installing tests ===\"\npsql -f /db/tests/install_tests.sql\n\necho \"=== Running tests ===\"\nRESULTS=$(psql -t -c \"SELECT * FROM test.run_all_tests();\")\necho \"$RESULTS\"\n\n# Check for failures\nif echo \"$RESULTS\" | grep -qE \"FAIL|ERROR\"; then\n echo \"=== TESTS FAILED ===\"\n exit 1\nfi\n\necho \"=== ALL TESTS PASSED ===\"\nexit 0\n```\n\n### Migration Deployment Script\n\n```bash\n#!/bin/bash\n# scripts/deploy-migrations.sh\n\nset -e\n\n# Configuration\nLOCK_TIMEOUT=${LOCK_TIMEOUT:-30}\nSTATEMENT_TIMEOUT=${STATEMENT_TIMEOUT:-300}\n\necho \"=== Starting migration deployment ===\"\necho \"Database: $PGDATABASE @ $PGHOST\"\n\n# Set timeouts\nexport PGOPTIONS=\"-c lock_timeout=${LOCK_TIMEOUT}s -c statement_timeout=${STATEMENT_TIMEOUT}s\"\n\n# Check migration system exists\nif ! psql -c \"SELECT 1 FROM app_migration.changelog LIMIT 1\" 2>/dev/null; then\n echo \"Installing migration system...\"\n psql -f db/scripts/001_install_migration_system.sql\n psql -f db/scripts/002_migration_runner_helpers.sql\nfi\n\n# Acquire migration lock\necho \"Acquiring migration lock...\"\npsql -c \"SELECT app_migration.acquire_lock();\"\n\n# Track if we need to release lock\nLOCK_ACQUIRED=true\ncleanup() {\n if [ \"$LOCK_ACQUIRED\" = true ]; then\n echo \"Releasing migration lock...\"\n psql -c \"SELECT app_migration.release_lock();\" || true\n fi\n}\ntrap cleanup EXIT\n\n# Get last applied version\nLAST_VERSION=$(psql -t -c \"\n SELECT COALESCE(MAX(version), '000') \n FROM app_migration.changelog \n WHERE type = 'versioned' AND success = true;\n\" | tr -d ' ')\n\necho \"Last applied version: $LAST_VERSION\"\n\n# Find and apply new migrations\nfor f in db/migrations/V*.sql; do\n if [ -f \"$f\" ]; then\n VERSION=$(basename \"$f\" | sed 's/V\$[0-9]*\$__.*/\\1/')\n if [ \"$VERSION\" -gt \"$LAST_VERSION\" ]; then\n echo \"Applying: $f\"\n psql -f \"$f\"\n else\n echo \"Skipping (already applied): $f\"\n fi\n fi\ndone\n\n# Apply repeatable migrations\necho \"Checking repeatable migrations...\"\nfor f in db/migrations/R__*.sql; do\n if [ -f \"$f\" ]; then\n FILENAME=$(basename \"$f\")\n CHECKSUM=$(md5sum \"$f\" | cut -d' ' -f1)\n \n LAST_CHECKSUM=$(psql -t -c \"\n SELECT checksum FROM app_migration.changelog \n WHERE filename = '$FILENAME' AND success = true\n ORDER BY executed_at DESC LIMIT 1;\n \" | tr -d ' ')\n \n if [ \"$CHECKSUM\" != \"$LAST_CHECKSUM\" ]; then\n echo \"Applying (changed): $f\"\n psql -f \"$f\"\n else\n echo \"Skipping (unchanged): $f\"\n fi\n fi\ndone\n\n# Release lock\necho \"Releasing migration lock...\"\npsql -c \"SELECT app_migration.release_lock();\"\nLOCK_ACQUIRED=false\n\necho \"=== Migration deployment complete ===\"\n```\n\n## Migration Validation\n\n### Pre-deployment Validation Script\n\n```bash\n#!/bin/bash\n# scripts/validate-migrations.sh\n\nset -e\n\nERRORS=0\n\necho \"=== Validating migrations ===\"\n\n# Check naming convention\necho \"Checking naming conventions...\"\nfor f in db/migrations/*.sql; do\n filename=$(basename \"$f\")\n \n # Versioned migrations\n if [[ $filename == V* ]]; then\n if ! [[ $filename =~ ^V[0-9]{3}__[a-z][a-z0-9_]*\\.sql$ ]]; then\n echo \"ERROR: Invalid versioned migration name: $filename\"\n echo \" Expected: V001__description_here.sql\"\n ((ERRORS++))\n fi\n fi\n \n # Repeatable migrations\n if [[ $filename == R__* ]]; then\n if ! [[ $filename =~ ^R__[a-z][a-z0-9_]*\\.sql$ ]]; then\n echo \"ERROR: Invalid repeatable migration name: $filename\"\n echo \" Expected: R__description.sql\"\n ((ERRORS++))\n fi\n fi\ndone\n\n# Check for sequential versioning\necho \"Checking version sequence...\"\nPREV_VERSION=0\nfor f in db/migrations/V*.sql; do\n if [ -f \"$f\" ]; then\n VERSION=$(basename \"$f\" | sed 's/V0*\$[0-9]*\$__.*/\\1/')\n EXPECTED=$((PREV_VERSION + 1))\n \n if [ \"$VERSION\" != \"$EXPECTED\" ]; then\n echo \"WARNING: Non-sequential version: $f (expected V$(printf '%03d' $EXPECTED))\"\n fi\n PREV_VERSION=$VERSION\n fi\ndone\n\n# Check for dangerous operations\necho \"Checking for dangerous operations...\"\nfor f in db/migrations/V*.sql; do\n if [ -f \"$f\" ]; then\n # Check for DROP TABLE without IF EXISTS\n if grep -qiE \"DROP\\s+TABLE\\s+(?!IF\\s+EXISTS)\" \"$f\"; then\n echo \"WARNING: $f contains DROP TABLE without IF EXISTS\"\n fi\n \n # Check for unqualified DELETE\n if grep -qiE \"DELETE\\s+FROM\\s+\\w+\\s*;\" \"$f\"; then\n echo \"WARNING: $f contains DELETE without WHERE clause\"\n fi\n \n # Check for TRUNCATE\n if grep -qi \"TRUNCATE\" \"$f\"; then\n echo \"WARNING: $f contains TRUNCATE\"\n fi\n fi\ndone\n\n# Syntax check against test database\necho \"Checking SQL syntax...\"\nfor f in db/migrations/*.sql; do\n if [ -f \"$f\" ]; then\n # Try to parse the SQL\n if ! psql -c \"\\\\set ON_ERROR_STOP on\" -c \"BEGIN;\" -f \"$f\" -c \"ROLLBACK;\" 2>/dev/null; then\n echo \"ERROR: Syntax error in $f\"\n ((ERRORS++))\n fi\n fi\ndone\n\nif [ $ERRORS -gt 0 ]; then\n echo \"=== VALIDATION FAILED: $ERRORS errors ===\"\n exit 1\nfi\n\necho \"=== VALIDATION PASSED ===\"\nexit 0\n```\n\n## Schema Comparison\n\n### Schema Diff Script\n\n```bash\n#!/bin/bash\n# scripts/schema-diff.sh\n# Compare schema between two databases\n\nSOURCE_DB=${1:-\"source_db\"}\nTARGET_DB=${2:-\"target_db\"}\n\necho \"Comparing $SOURCE_DB -> $TARGET_DB\"\n\n# Dump schemas\npg_dump -s -d \"$SOURCE_DB\" > /tmp/source_schema.sql\npg_dump -s -d \"$TARGET_DB\" > /tmp/target_schema.sql\n\n# Compare\ndiff -u /tmp/source_schema.sql /tmp/target_schema.sql > /tmp/schema_diff.txt || true\n\nif [ -s /tmp/schema_diff.txt ]; then\n echo \"Schema differences found:\"\n cat /tmp/schema_diff.txt\n exit 1\nelse\n echo \"Schemas are identical\"\n exit 0\nfi\n```\n\n### Schema Snapshot Function\n\n```sql\n-- Create schema snapshot for comparison\nCREATE OR REPLACE FUNCTION api.get_schema_snapshot()\nRETURNS jsonb\nLANGUAGE sql\nSTABLE\nAS $\n SELECT jsonb_build_object(\n 'tables', (\n SELECT jsonb_agg(jsonb_build_object(\n 'schema', table_schema,\n 'name', table_name,\n 'columns', (\n SELECT jsonb_agg(jsonb_build_object(\n 'name', column_name,\n 'type', data_type,\n 'nullable', is_nullable\n ) ORDER BY ordinal_position)\n FROM information_schema.columns c\n WHERE c.table_schema = t.table_schema \n AND c.table_name = t.table_name\n )\n ))\n FROM information_schema.tables t\n WHERE table_schema IN ('data', 'api', 'private')\n AND table_type = 'BASE TABLE'\n ),\n 'functions', (\n SELECT jsonb_agg(jsonb_build_object(\n 'schema', n.nspname,\n 'name', p.proname,\n 'args', pg_get_function_arguments(p.oid)\n ))\n FROM pg_proc p\n JOIN pg_namespace n ON p.pronamespace = n.oid\n WHERE n.nspname IN ('api', 'private')\n ),\n 'snapshot_at', now()\n );\n$;\n```\n\n## Deployment Strategies\n\n### Blue-Green Schema Deployment\n\n```sql\n-- Use schema versioning for blue-green deployments\n\n-- Current production uses 'api_v1' schema\n-- Deploy new version to 'api_v2' schema\n\n-- 1. Create new schema\nCREATE SCHEMA api_v2;\n\n-- 2. Deploy new functions to api_v2\nCREATE FUNCTION api_v2.get_customer(...) ...;\n\n-- 3. Test api_v2 thoroughly\n\n-- 4. Switch traffic (update search_path)\nALTER ROLE app_service SET search_path = api_v2, pg_temp;\n\n-- 5. After verification, drop old schema\nDROP SCHEMA api_v1 CASCADE;\nALTER SCHEMA api_v2 RENAME TO api;\n```\n\n### Canary Deployment\n\n```sql\n-- Route percentage of traffic to new version\nCREATE OR REPLACE FUNCTION api.get_customer(in_id uuid)\nRETURNS TABLE (...)\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- 10% canary\n IF random() \u003c 0.10 THEN\n RETURN QUERY SELECT * FROM api_v2.get_customer_impl(in_id);\n ELSE\n RETURN QUERY SELECT * FROM api_v1.get_customer_impl(in_id);\n END IF;\nEND;\n$;\n```\n\n### Rollback Procedure\n\n```bash\n#!/bin/bash\n# scripts/rollback-migration.sh\n\nVERSION=$1\n\nif [ -z \"$VERSION\" ]; then\n echo \"Usage: $0 \u003cversion>\"\n exit 1\nfi\n\necho \"Rolling back to version $VERSION...\"\n\n# Get rollback SQL\nROLLBACK_SQL=$(psql -t -c \"\n SELECT rollback_sql \n FROM app_migration.rollback_scripts \n WHERE version = '$VERSION';\n\")\n\nif [ -z \"$ROLLBACK_SQL\" ]; then\n echo \"ERROR: No rollback script found for version $VERSION\"\n exit 1\nfi\n\n# Confirm\nread -p \"Are you sure you want to rollback? (yes/no): \" CONFIRM\nif [ \"$CONFIRM\" != \"yes\" ]; then\n echo \"Rollback cancelled\"\n exit 0\nfi\n\n# Execute rollback\necho \"Executing rollback...\"\npsql -c \"CALL app_migration.rollback('$VERSION');\"\n\necho \"Rollback complete\"\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":21235,"content_sha256":"ccaf770c9b7e6bf43ac09ccc04b01b7735b467ab7976bf21f21fbe715d23a6c0"},{"filename":"references/coding-standards-trivadis.md","content":"# PL/pgSQL Coding Standards (Trivadis-Style)\n\nThis document adapts the well-established [Trivadis PL/SQL & SQL Coding Guidelines v4.4](https://trivadis.github.io/plsql-and-sql-coding-guidelines/v4.4/) for PostgreSQL PL/pgSQL development. These conventions provide a consistent, readable, and maintainable codebase familiar to developers coming from Oracle backgrounds.\n\n## Table of Contents\n\n1. [Naming Conventions](#naming-conventions)\n2. [Variable and Parameter Naming](#variable-and-parameter-naming)\n3. [Code Structure](#code-structure)\n4. [Package-Like Organization](#package-like-organization)\n5. [Error Handling](#error-handling)\n6. [SQL Guidelines](#sql-guidelines)\n7. [Documentation Standards](#documentation-standards)\n8. [Complete Example](#complete-example)\n\n## Naming Conventions\n\n### General Guidelines\n\n1. **Always use lowercase** - PostgreSQL folds unquoted identifiers to lowercase\n2. **Never use double-quoted identifiers** - Avoid case sensitivity issues\n3. **Use meaningful, specific names** - Self-documenting code\n4. **Avoid abbreviations** unless widely known and accepted\n5. **Keep abbreviations under 5 characters**\n6. **Never use PostgreSQL reserved words** as identifiers\n7. **Use one spoken language** (e.g., English) consistently\n8. **Use snake_case** for all identifiers\n\n### PL/pgSQL Identifier Prefixes\n\nFollow the `{prefix}_{content}` pattern for variables and parameters:\n\n```mermaid\ngraph LR\n subgraph \"Variable Prefixes\"\n G[\"g_ = Global/Session\"]\n L[\"l_ = Local\"]\n CO[\"co_ = Constant\"]\n end\n \n subgraph \"Parameter Prefixes\"\n IN[\"in_ = Input\"]\n OUT[\"out_ = Output\"]\n IO[\"io_ = Input/Output\"]\n end\n \n subgraph \"Other Prefixes\"\n C[\"c_ = Cursor\"]\n R[\"r_ = Record\"]\n T[\"t_ = Array/Table type\"]\n E[\"e_ = Exception\"]\n end\n \n style G fill:#c8e6c9\n style L fill:#c8e6c9\n style CO fill:#c8e6c9\n style IN fill:#bbdefb\n style OUT fill:#bbdefb\n style IO fill:#bbdefb\n```\n\n| Identifier Type | Prefix | Suffix | Example |\n|-----------------|--------|--------|---------|\n| **Global/Session Variable** | `g_` | | `g_current_user_id` |\n| **Local Variable** | `l_` | | `l_customer_count` |\n| **Constant** | `co_` | | `co_max_retry_count` |\n| **Cursor** | `c_` | | `c_active_orders` |\n| **Record** | `r_` | | `r_customer` |\n| **Array/Table Variable** | `t_` | | `t_order_ids` |\n| **Record Type** | `r_` | `_type` | `r_customer_type` |\n| **Array Type** | `t_` | `_type` | `t_order_ids_type` |\n| **Exception** | `e_` | | `e_customer_not_found` |\n| **IN Parameter** | `in_` | | `in_customer_id` |\n| **OUT Parameter** (functions only) | `out_` | | `out_order_total` |\n| **INOUT Parameter** (procedures) | `io_` | | `io_id` |\n\n> **PostgreSQL Note**: Procedures only support INOUT parameters (no OUT). Use `io_` prefix for all INOUT parameters in procedures, even if they're semantically output-only.\n\n### Database Object Naming\n\n| Object Type | Convention | Example |\n|-------------|------------|---------|\n| **Schema** | `snake_case` | `data`, `api`, `private` |\n| **Table** | Plural, `snake_case` | `customers`, `order_items` |\n| **Column** | Singular, `snake_case` | `customer_id`, `created_at` |\n| **Primary Key** | `{table}_pk` | `customers_pk` |\n| **Foreign Key** | `{table}_{reftable}_fk` | `orders_customers_fk` |\n| **Unique Constraint** | `{table}_{columns}_uk` | `customers_email_uk` |\n| **Check Constraint** | `{table}_{column}_ck` | `orders_status_ck` |\n| **Index** | `{table}_{columns}_idx` | `orders_customer_id_idx` |\n| **Unique Index** | `{table}_{columns}_key` | `customers_email_key` |\n| **Function** | `{verb}_{noun}` or `{noun}_by_{filter}` | `calculate_total`, `customer_by_email` |\n| **Procedure** | `{verb}_{noun}` | `insert_customer`, `update_status` |\n| **Trigger** | `{table}_{timing}{event}_trg` | `orders_biu_trg` |\n| **View** | Descriptive or `v_{name}` | `active_customers`, `v_order_summary` |\n| **Sequence** | `{table}_seq` | `customers_seq` |\n| **Type** | `{name}_type` | `address_type` |\n\n## Variable and Parameter Naming\n\n### Local Variables (l_ prefix)\n\n```sql\nCREATE FUNCTION api.calculate_order_total(in_order_id uuid)\nRETURNS numeric\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n -- Local variables with l_ prefix\n l_subtotal numeric;\n l_tax_rate numeric;\n l_discount numeric;\n l_total numeric;\n l_customer_id uuid;\nBEGIN\n -- Get order details\n SELECT subtotal, tax_rate, customer_id\n INTO l_subtotal, l_tax_rate, l_customer_id\n FROM data.orders\n WHERE id = in_order_id;\n \n -- Calculate discount based on customer\n l_discount := private.get_customer_discount(l_customer_id);\n \n -- Calculate total\n l_total := l_subtotal * (1 - l_discount) * (1 + l_tax_rate);\n \n RETURN l_total;\nEND;\n$;\n```\n\n### Constants (co_ prefix)\n\n```sql\nCREATE FUNCTION api.validate_order(in_order_id uuid)\nRETURNS boolean\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n -- Constants with co_ prefix\n co_max_order_amount CONSTANT numeric := 100000.00;\n co_min_order_amount CONSTANT numeric := 0.01;\n co_max_items_per_order CONSTANT integer := 100;\n \n -- Local variables\n l_order_total numeric;\n l_item_count integer;\nBEGIN\n SELECT total, item_count\n INTO l_order_total, l_item_count\n FROM data.orders o\n JOIN (SELECT order_id, COUNT(*) as item_count \n FROM data.order_items GROUP BY order_id) oi \n ON oi.order_id = o.id\n WHERE o.id = in_order_id;\n \n -- Validate against constants\n IF l_order_total > co_max_order_amount THEN\n RETURN false;\n END IF;\n \n IF l_order_total \u003c co_min_order_amount THEN\n RETURN false;\n END IF;\n \n IF l_item_count > co_max_items_per_order THEN\n RETURN false;\n END IF;\n \n RETURN true;\nEND;\n$;\n```\n\n### Session/Global Variables (g_ prefix)\n\nPostgreSQL doesn't have true package-level global variables like Oracle. Instead, use:\n\n1. **Session variables** via `SET` and `current_setting()`\n2. **Configuration parameters** for application-wide settings\n\n```sql\n-- Setting a session variable\nSET myapp.current_user_id = 'user-uuid-here';\nSET myapp.current_tenant_id = 'tenant-uuid-here';\n\n-- Reading session variables in functions\nCREATE FUNCTION private.get_current_user_id()\nRETURNS uuid\nLANGUAGE sql\nSTABLE\nAS $\n SELECT NULLIF(current_setting('myapp.current_user_id', true), '')::uuid;\n$;\n\n-- Using in a function with g_ prefix for clarity\nCREATE FUNCTION api.get_my_orders()\nRETURNS TABLE (id uuid, total numeric, status text)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n g_current_user_id uuid; -- \"global\" from session context\nBEGIN\n -- Get session variable\n g_current_user_id := private.get_current_user_id();\n \n IF g_current_user_id IS NULL THEN\n RAISE EXCEPTION 'No user context set'\n USING ERRCODE = 'P0001';\n END IF;\n \n RETURN QUERY\n SELECT o.id, o.total, o.status\n FROM data.orders o\n WHERE o.customer_id = g_current_user_id\n ORDER BY o.created_at DESC;\nEND;\n$;\n```\n\n### Records (r_ prefix)\n\n```sql\nCREATE PROCEDURE api.process_order(in_order_id uuid)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n r_order data.orders%ROWTYPE; -- Record matching table structure\n r_customer data.customers%ROWTYPE;\nBEGIN\n -- Fetch into record\n SELECT * INTO r_order\n FROM data.orders\n WHERE id = in_order_id;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Order not found: %', in_order_id\n USING ERRCODE = 'P0002';\n END IF;\n \n -- Fetch related customer\n SELECT * INTO r_customer\n FROM data.customers\n WHERE id = r_order.customer_id;\n \n -- Use record fields\n IF r_order.status = 'pending' AND r_customer.is_active THEN\n UPDATE data.orders\n SET status = 'processing'\n WHERE id = in_order_id;\n END IF;\nEND;\n$;\n```\n\n### Cursors (c_ prefix)\n\n```sql\nCREATE PROCEDURE api.batch_process_orders(in_status text)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n c_orders CURSOR FOR\n SELECT id, customer_id, total\n FROM data.orders\n WHERE status = in_status\n ORDER BY created_at;\n \n r_order RECORD;\n l_processed_count integer := 0;\nBEGIN\n FOR r_order IN c_orders LOOP\n -- Process each order\n PERFORM private.process_single_order(r_order.id);\n l_processed_count := l_processed_count + 1;\n \n -- Commit every 100 records (if needed)\n IF l_processed_count % 100 = 0 THEN\n RAISE NOTICE 'Processed % orders', l_processed_count;\n END IF;\n END LOOP;\n \n RAISE NOTICE 'Total orders processed: %', l_processed_count;\nEND;\n$;\n```\n\n### Arrays (t_ prefix)\n\n```sql\nCREATE FUNCTION api.get_customers_by_ids(in_ids uuid[])\nRETURNS TABLE (id uuid, email text, name text)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n t_valid_ids uuid[]; -- Array variable with t_ prefix\n l_count integer;\nBEGIN\n -- Filter to only active customer IDs\n SELECT array_agg(c.id)\n INTO t_valid_ids\n FROM data.customers c\n WHERE c.id = ANY(in_ids)\n AND c.is_active = true;\n \n l_count := COALESCE(array_length(t_valid_ids, 1), 0);\n RAISE NOTICE 'Found % valid customers out of % requested', \n l_count, array_length(in_ids, 1);\n \n RETURN QUERY\n SELECT c.id, c.email, c.name\n FROM data.customers c\n WHERE c.id = ANY(t_valid_ids);\nEND;\n$;\n```\n\n### Parameters (in_, out_, io_ prefixes)\n\n```sql\n-- IN parameters (read-only input)\nCREATE FUNCTION api.get_customer(in_customer_id uuid)\nRETURNS TABLE (id uuid, email text, name text)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name\n FROM data.customers\n WHERE id = in_customer_id;\n$;\n\n-- INOUT parameters (input that also returns output)\nCREATE PROCEDURE api.insert_customer(\n in_email text,\n in_name text,\n INOUT io_id uuid DEFAULT NULL -- Returns generated ID\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.customers (email, name)\n VALUES (lower(trim(in_email)), trim(in_name))\n RETURNING id INTO io_id;\nEND;\n$;\n\n-- Multiple output values using INOUT (PostgreSQL has no OUT for procedures)\nCREATE PROCEDURE api.get_order_summary(\n in_order_id uuid,\n INOUT io_total numeric DEFAULT NULL,\n INOUT io_item_count integer DEFAULT NULL,\n INOUT io_status text DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n SELECT o.total, COUNT(oi.id), o.status\n INTO io_total, io_item_count, io_status\n FROM data.orders o\n LEFT JOIN data.order_items oi ON oi.order_id = o.id\n WHERE o.id = in_order_id\n GROUP BY o.id;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Order not found: %', in_order_id\n USING ERRCODE = 'P0002';\n END IF;\nEND;\n$;\n```\n\n## Code Structure\n\n### Indentation Standard\n\nUse **3-space indentation** for PL/pgSQL code blocks (Trivadis v4.4 standard):\n\n```sql\nCREATE FUNCTION api.example_function(in_id uuid)\nRETURNS text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_result text; -- 3 spaces\nBEGIN\n IF in_id IS NOT NULL THEN\n SELECT name INTO l_result -- 6 spaces\n FROM data.customers\n WHERE id = in_id;\n END IF;\n\n RETURN l_result;\nEND example_function;\n$;\n```\n\n> **Note**: While 2-space and 4-space indentation are common in other languages, 3-space indentation is the Trivadis standard for PL/SQL and PL/pgSQL. This provides good readability while maintaining compact code.\n\n### Function/Procedure Template\n\n```sql\n-- ============================================================================\n-- Function: api.function_name\n-- Purpose: Brief description of what this function does\n-- Parameters:\n-- in_param1 - Description of first parameter\n-- in_param2 - Description of second parameter (optional)\n-- Returns: Description of return value\n-- Raises: P0002 - When resource not found\n-- P0001 - When business rule violated\n-- ============================================================================\nCREATE OR REPLACE FUNCTION api.function_name(\n in_param1 uuid,\n in_param2 text DEFAULT NULL\n)\nRETURNS return_type\nLANGUAGE plpgsql\nSTABLE -- or VOLATILE for modifications\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n -- Constants (co_ prefix)\n co_max_value CONSTANT integer := 100;\n \n -- Local variables (l_ prefix)\n l_result return_type;\n l_temp integer;\nBEGIN\n -- Input validation\n IF in_param1 IS NULL THEN\n RAISE EXCEPTION 'Parameter in_param1 cannot be NULL'\n USING ERRCODE = 'P0001';\n END IF;\n \n -- Main logic\n -- ...\n \n RETURN l_result;\n \nEXCEPTION\n WHEN no_data_found THEN\n RAISE EXCEPTION 'Resource not found: %', in_param1\n USING ERRCODE = 'P0002';\n WHEN OTHERS THEN\n RAISE EXCEPTION 'Unexpected error in function_name: %', SQLERRM\n USING ERRCODE = 'P0003';\nEND;\n$;\n\nCOMMENT ON FUNCTION api.function_name(uuid, text) IS \n 'Brief description for documentation';\n```\n\n### Block Labels\n\nLabel nested blocks for clarity (Trivadis guideline G-1010):\n\n```sql\nCREATE PROCEDURE api.complex_operation(in_order_id uuid)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_status text;\nBEGIN\n -- Get current status\n SELECT status INTO l_status\n FROM data.orders WHERE id = in_order_id;\n \n \u003c\u003cvalidation_block>>\n BEGIN\n IF l_status IS NULL THEN\n RAISE EXCEPTION 'Order not found';\n END IF;\n \n IF l_status = 'cancelled' THEN\n RAISE EXCEPTION 'Cannot process cancelled order';\n END IF;\n END validation_block;\n \n \u003c\u003cprocessing_block>>\n DECLARE\n l_item_count integer;\n BEGIN\n SELECT COUNT(*) INTO l_item_count\n FROM data.order_items\n WHERE order_id = in_order_id;\n \n IF l_item_count = 0 THEN\n RAISE EXCEPTION 'Order has no items';\n END IF;\n \n -- Process the order\n UPDATE data.orders\n SET status = 'processing'\n WHERE id = in_order_id;\n END processing_block;\nEND;\n$;\n```\n\n### END Labels (G-7120)\n\nAlways add the function or procedure name after the END keyword for clarity:\n\n```sql\n-- Good (Trivadis G-7120)\nCREATE FUNCTION api.calculate_order_total(in_order_id uuid)\nRETURNS numeric\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- function logic...\n RETURN l_total;\nEND calculate_order_total;\n$;\n\n-- Avoid (unnamed END)\nCREATE FUNCTION api.calculate_order_total(in_order_id uuid)\nRETURNS numeric\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- function logic...\n RETURN l_total;\nEND;\n$;\n```\n\nThis convention:\n- Improves readability in long procedures\n- Makes it clear which block is ending\n- Helps identify mismatched BEGIN/END pairs\n- Aligns with Oracle PL/SQL best practices\n\n### Loop Labels\n\nAlways label loops (Trivadis guideline G-4320):\n\n```sql\nCREATE PROCEDURE api.process_all_pending_orders()\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n r_order RECORD;\n l_success_count integer := 0;\n l_error_count integer := 0;\nBEGIN\n \u003c\u003corder_loop>>\n FOR r_order IN \n SELECT id, customer_id \n FROM data.orders \n WHERE status = 'pending'\n LOOP\n \u003c\u003cprocess_block>>\n BEGIN\n PERFORM private.process_order(r_order.id);\n l_success_count := l_success_count + 1;\n EXCEPTION\n WHEN OTHERS THEN\n l_error_count := l_error_count + 1;\n RAISE WARNING 'Failed to process order %: %', \n r_order.id, SQLERRM;\n CONTINUE order_loop; -- Skip to next iteration\n END process_block;\n END LOOP order_loop;\n \n RAISE NOTICE 'Processed: % success, % errors', \n l_success_count, l_error_count;\nEND;\n$;\n```\n\n## Package-Like Organization\n\n> **Important**: This section describes how to achieve Oracle package-like organization while **preserving** the existing schema separation pattern (`data`/`private`/`api`).\n\n### Understanding the Challenge\n\nOracle packages provide:\n1. **Namespace grouping** - Related procedures/functions together\n2. **Public vs Private** - Specification (public) vs Body (private)\n3. **Session state** - Package-level variables\n4. **Initialization** - One-time setup code\n\nPostgreSQL schemas already provide #1 and #2 through our `api`/`private` separation. We need patterns for #3 and #4.\n\n### Recommended Approach: Functional Grouping Within Schemas\n\n```mermaid\nflowchart TB\n subgraph \"Oracle Package Approach\"\n PKG[\"employees_pkg\u003cbr/>━━━━━━━━━━━━━\u003cbr/>• get_employee (public)\u003cbr/>• insert_employee (public)\u003cbr/>• validate_email (private)\u003cbr/>• g_cache (variable)\"]\n end\n \n subgraph \"PostgreSQL Schema Approach\"\n subgraph API[\"api schema (Public Interface)\"]\n API_EMP[\"employees module\u003cbr/>━━━━━━━━━━━━━\u003cbr/>api.emp_get()\u003cbr/>api.emp_insert()\u003cbr/>api.emp_select()\"]\n API_ORD[\"orders module\u003cbr/>━━━━━━━━━━━━━\u003cbr/>api.ord_get()\u003cbr/>api.ord_insert()\"]\n end\n \n subgraph PRIVATE[\"private schema (Private Implementation)\"]\n PRIV_EMP[\"employees helpers\u003cbr/>━━━━━━━━━━━━━\u003cbr/>private.emp_validate_email()\u003cbr/>private.emp_hash_password()\"]\n PRIV_ORD[\"orders helpers\u003cbr/>━━━━━━━━━━━━━\u003cbr/>private.ord_calculate_total()\"]\n end\n end\n \n PKG -->|\"maps to\"| API_EMP\n PKG -->|\"maps to\"| PRIV_EMP\n \n style PKG fill:#fff3e0\n style API fill:#c8e6c9\n style PRIVATE fill:#ffe0b2\n```\n\n### Option 1: Prefix-Based Grouping (Recommended)\n\nGroup related functions using a consistent prefix within the existing schemas:\n\n```sql\n-- ============================================================================\n-- \"employees\" module - Public API (in api schema)\n-- ============================================================================\nCREATE FUNCTION api.emp_get(in_id uuid) ...;\nCREATE FUNCTION api.emp_get_by_email(in_email text) ...;\nCREATE FUNCTION api.emp_select(in_is_active boolean DEFAULT NULL) ...;\nCREATE PROCEDURE api.emp_insert(in_email text, in_name text, INOUT io_id uuid) ...;\nCREATE PROCEDURE api.emp_update(in_id uuid, in_name text DEFAULT NULL) ...;\nCREATE PROCEDURE api.emp_delete(in_id uuid) ...;\n\n-- ============================================================================\n-- \"employees\" module - Private helpers (in private schema)\n-- ============================================================================\nCREATE FUNCTION private.emp_validate_email(in_email text) ...;\nCREATE FUNCTION private.emp_hash_password(in_password text) ...;\nCREATE FUNCTION private.emp_check_permissions(in_emp_id uuid, in_action text) ...;\n\n-- ============================================================================\n-- \"orders\" module - Public API (in api schema)\n-- ============================================================================\nCREATE FUNCTION api.ord_get(in_id uuid) ...;\nCREATE FUNCTION api.ord_select_by_customer(in_customer_id uuid) ...;\nCREATE PROCEDURE api.ord_insert(in_customer_id uuid, INOUT io_id uuid) ...;\nCREATE PROCEDURE api.ord_update_status(in_id uuid, in_status text) ...;\n\n-- ============================================================================\n-- \"orders\" module - Private helpers (in private schema)\n-- ============================================================================\nCREATE FUNCTION private.ord_calculate_total(in_order_id uuid) ...;\nCREATE FUNCTION private.ord_validate_status_transition(in_from text, in_to text) ...;\n```\n\n**Benefits:**\n- Maintains `data`/`private`/`api` separation ✓\n- Easy to find related functions (alphabetical sorting groups them)\n- Clear public/private distinction via schema\n- No additional schema complexity\n\n### Option 2: Sub-Schema Pattern (For Large Applications)\n\n> **For Oracle migrators:** the full walkthrough — file-per-module layout, per-package grants, IDE navigation, and design rules — is in [oracle-migration-guide.md → Packages to Schemas → Oracle-Package Style](oracle-migration-guide.md#oracle-package-style-sub-schemas--file-per-module). The brief overview below stands; the Oracle guide is the long-form companion.\n\nFor very large applications, create sub-schemas under `api`:\n\n```sql\n-- Create sub-schemas for major modules\nCREATE SCHEMA api_employees;\nCREATE SCHEMA api_orders;\nCREATE SCHEMA api_inventory;\n\n-- Keep shared private schema, or split it too\nCREATE SCHEMA private_employees;\nCREATE SCHEMA private_orders;\n```\n\n**Schema Structure:**\n```\napi_employees -- Public employee functions\napi_orders -- Public order functions\napi_inventory -- Public inventory functions\nprivate_employees -- Private employee helpers\nprivate_orders -- Private order helpers\nprivate -- Shared private utilities\ndata -- All tables (unchanged)\n```\n\n> ⚠️ **Warning**: This adds complexity. Only use for very large codebases (50+ tables, 200+ functions).\n\n### Option 3: Nested Function Approach (Package Body Simulation)\n\nFor complex modules needing private helper functions visible only within the module:\n\n```sql\nCREATE FUNCTION api.emp_insert(\n in_email text,\n in_name text,\n in_password text,\n INOUT io_id uuid DEFAULT NULL\n)\nRETURNS uuid\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_password_hash text;\n l_email_valid boolean;\n \n -- ========================================\n -- Private nested function (like package body)\n -- ========================================\n -- Note: PL/pgSQL doesn't support nested functions directly.\n -- Use private schema functions instead:\n -- l_password_hash := private.emp_hash_password(in_password);\n -- l_email_valid := private.emp_validate_email(in_email);\nBEGIN\n -- Validate email using private helper\n IF NOT private.emp_validate_email(in_email) THEN\n RAISE EXCEPTION 'Invalid email format: %', in_email\n USING ERRCODE = 'P0001';\n END IF;\n \n -- Hash password using private helper\n l_password_hash := private.emp_hash_password(in_password);\n \n -- Insert employee\n INSERT INTO data.employees (email, name, password_hash)\n VALUES (lower(trim(in_email)), trim(in_name), l_password_hash)\n RETURNING id INTO io_id;\nEND;\n$;\n```\n\n### Package Initialization Equivalent\n\nOracle packages can have initialization code that runs once per session. In PostgreSQL:\n\n```sql\n-- Create an initialization function\nCREATE FUNCTION private.init_app_context()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Set session-level configuration\n PERFORM set_config('myapp.initialized', 'true', false);\n PERFORM set_config('myapp.init_time', now()::text, false);\n \n -- Any other one-time setup\n RAISE NOTICE 'Application context initialized at %', now();\nEND;\n$;\n\n-- Check if initialization needed in other functions\nCREATE FUNCTION private.ensure_initialized()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n IF current_setting('myapp.initialized', true) IS DISTINCT FROM 'true' THEN\n PERFORM private.init_app_context();\n END IF;\nEND;\n$;\n\n-- Use in API functions if needed\nCREATE FUNCTION api.some_function(in_param text)\nRETURNS text\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n -- Ensure app is initialized\n PERFORM private.ensure_initialized();\n \n -- ... rest of function\n RETURN in_param;\nEND;\n$;\n```\n\n### Package Constants Equivalent\n\n```sql\n-- Create a \"constants\" function that returns a record\nCREATE FUNCTION private.co_app_constants()\nRETURNS TABLE (\n max_login_attempts integer,\n session_timeout_min integer,\n default_page_size integer,\n app_version text\n)\nLANGUAGE sql\nIMMUTABLE\nPARALLEL SAFE\nAS $\n SELECT \n 5::integer, -- max_login_attempts\n 30::integer, -- session_timeout_min\n 25::integer, -- default_page_size\n '2.1.0'::text; -- app_version\n$;\n\n-- Usage in other functions\nCREATE FUNCTION api.get_paginated_customers(\n in_page integer DEFAULT 1,\n in_size integer DEFAULT NULL\n)\nRETURNS TABLE (id uuid, email text, name text)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_page_size integer;\nBEGIN\n -- Get default from constants if not provided\n l_page_size := COALESCE(\n in_size, \n (SELECT default_page_size FROM private.co_app_constants())\n );\n \n RETURN QUERY\n SELECT c.id, c.email, c.name\n FROM data.customers c\n ORDER BY c.created_at DESC\n LIMIT l_page_size\n OFFSET (in_page - 1) * l_page_size;\nEND;\n$;\n```\n\n## Error Handling\n\n### Named Exceptions (e_ prefix)\n\n```sql\nCREATE PROCEDURE api.transfer_funds(\n in_from_account_id uuid,\n in_to_account_id uuid,\n in_amount numeric\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n -- Named exception conditions\n e_insufficient_funds CONSTANT text := 'P0010';\n e_account_not_found CONSTANT text := 'P0011';\n e_account_inactive CONSTANT text := 'P0012';\n e_self_transfer CONSTANT text := 'P0013';\n \n l_from_balance numeric;\n l_from_active boolean;\n l_to_active boolean;\nBEGIN\n -- Validate: not self-transfer\n IF in_from_account_id = in_to_account_id THEN\n RAISE EXCEPTION 'Cannot transfer to same account'\n USING ERRCODE = e_self_transfer;\n END IF;\n \n -- Get source account\n SELECT balance, is_active \n INTO l_from_balance, l_from_active\n FROM data.accounts\n WHERE id = in_from_account_id\n FOR UPDATE;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Source account not found: %', in_from_account_id\n USING ERRCODE = e_account_not_found;\n END IF;\n \n IF NOT l_from_active THEN\n RAISE EXCEPTION 'Source account is inactive'\n USING ERRCODE = e_account_inactive;\n END IF;\n \n IF l_from_balance \u003c in_amount THEN\n RAISE EXCEPTION 'Insufficient funds: have %, need %', \n l_from_balance, in_amount\n USING ERRCODE = e_insufficient_funds;\n END IF;\n \n -- Validate destination\n SELECT is_active INTO l_to_active\n FROM data.accounts\n WHERE id = in_to_account_id\n FOR UPDATE;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Destination account not found: %', in_to_account_id\n USING ERRCODE = e_account_not_found;\n END IF;\n \n IF NOT l_to_active THEN\n RAISE EXCEPTION 'Destination account is inactive'\n USING ERRCODE = e_account_inactive;\n END IF;\n \n -- Perform transfer\n UPDATE data.accounts SET balance = balance - in_amount \n WHERE id = in_from_account_id;\n \n UPDATE data.accounts SET balance = balance + in_amount \n WHERE id = in_to_account_id;\nEND;\n$;\n```\n\n### Error Code Ranges\n\nDefine consistent error code ranges for your application:\n\n| Range | Category | Example |\n|-------|----------|---------|\n| `P0001-P0009` | General validation errors | `P0001` = Invalid input |\n| `P0010-P0019` | Account/Finance errors | `P0010` = Insufficient funds |\n| `P0020-P0029` | Order processing errors | `P0020` = Invalid status transition |\n| `P0030-P0039` | Customer errors | `P0030` = Customer not found |\n| `P0040-P0049` | Authentication errors | `P0040` = Invalid credentials |\n| `P0050-P0059` | Authorization errors | `P0050` = Permission denied |\n\n## SQL Guidelines\n\n### Always Specify Column Names (G-3110)\n\n```sql\n-- BAD: Using SELECT *\nINSERT INTO data.audit_log SELECT * FROM data.orders WHERE id = in_order_id;\n\n-- GOOD: Explicit columns\nINSERT INTO data.audit_log (order_id, order_total, order_status, logged_at)\nSELECT id, total, status, now()\nFROM data.orders\nWHERE id = in_order_id;\n```\n\n### Always Use Table Aliases (G-3120)\n\n```sql\n-- BAD: No aliases\nSELECT customers.id, orders.total\nFROM data.customers, data.orders\nWHERE customers.id = orders.customer_id;\n\n-- GOOD: With aliases\nSELECT c.id, o.total\nFROM data.customers c\nJOIN data.orders o ON o.customer_id = c.id;\n```\n\n### Use ANSI SQL-92 Joins (G-3130)\n\n```sql\n-- BAD: Old-style join\nSELECT c.name, o.total\nFROM data.customers c, data.orders o\nWHERE c.id = o.customer_id;\n\n-- GOOD: ANSI join syntax\nSELECT c.name, o.total\nFROM data.customers c\nINNER JOIN data.orders o ON o.customer_id = c.id;\n```\n\n### Use COALESCE Instead of Multiple NVL/NULLIF\n\n```sql\n-- Use COALESCE for NULL handling\nSELECT COALESCE(nickname, first_name, 'Guest') AS display_name\nFROM data.users;\n\n-- PostgreSQL CASE instead of Oracle DECODE\nSELECT \n CASE status\n WHEN 'A' THEN 'Active'\n WHEN 'I' THEN 'Inactive'\n WHEN 'P' THEN 'Pending'\n ELSE 'Unknown'\n END AS status_name\nFROM data.customers;\n```\n\n## Documentation Standards\n\n### Function Header Comments\n\n```sql\n-- ============================================================================\n-- Function: api.calculate_shipping_cost\n-- \n-- Purpose: Calculates shipping cost based on order weight, destination,\n-- and shipping method. Applies customer discounts if applicable.\n--\n-- Parameters:\n-- in_order_id UUID of the order to calculate shipping for\n-- in_shipping_method Shipping method code ('standard', 'express', 'overnight')\n--\n-- Returns: Calculated shipping cost as numeric(10,2)\n--\n-- Raises:\n-- P0002 - Order not found\n-- P0020 - Invalid shipping method\n-- P0021 - Shipping not available to destination\n--\n-- Example:\n-- SELECT api.calculate_shipping_cost('order-uuid', 'standard');\n--\n-- History:\n-- 2024-01-15 - Created (jsmith)\n-- 2024-02-20 - Added express shipping support (jdoe)\n-- ============================================================================\nCREATE OR REPLACE FUNCTION api.calculate_shipping_cost(\n in_order_id uuid,\n in_shipping_method text DEFAULT 'standard'\n)\nRETURNS numeric\nLANGUAGE plpgsql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n-- ... implementation\n$;\n```\n\n### Add Database Comments\n\n```sql\n-- Table comments\nCOMMENT ON TABLE data.customers IS \n 'Customer accounts with authentication and contact information';\n\n-- Column comments\nCOMMENT ON COLUMN data.customers.password_hash IS \n 'Bcrypt hash of customer password. Never expose via API.';\n\nCOMMENT ON COLUMN data.customers.is_active IS \n 'Soft delete flag. False = customer is deactivated.';\n\n-- Function comments\nCOMMENT ON FUNCTION api.emp_get(uuid) IS \n 'Retrieves employee by ID. Returns NULL if not found or inactive.';\n```\n\n## Complete Example\n\nHere's a complete \"orders\" module following all conventions:\n\n```sql\n-- ============================================================================\n-- ORDERS MODULE - Data Types\n-- ============================================================================\n\n-- Custom type for order status transitions\nCREATE TYPE private.order_status_type AS ENUM (\n 'draft', 'pending', 'confirmed', 'processing', \n 'shipped', 'delivered', 'cancelled'\n);\n\n-- ============================================================================\n-- ORDERS MODULE - Private Helper Functions\n-- ============================================================================\n\nCREATE OR REPLACE FUNCTION private.ord_validate_status_transition(\n in_current_status text,\n in_new_status text\n)\nRETURNS boolean\nLANGUAGE plpgsql\nIMMUTABLE\nAS $\nDECLARE\n co_valid_transitions CONSTANT jsonb := '{\n \"draft\": [\"pending\", \"cancelled\"],\n \"pending\": [\"confirmed\", \"cancelled\"],\n \"confirmed\": [\"processing\", \"cancelled\"],\n \"processing\": [\"shipped\", \"cancelled\"],\n \"shipped\": [\"delivered\"],\n \"delivered\": [],\n \"cancelled\": []\n }'::jsonb;\n \n t_allowed_statuses text[];\nBEGIN\n SELECT array_agg(value::text)\n INTO t_allowed_statuses\n FROM jsonb_array_elements_text(\n co_valid_transitions -> in_current_status\n );\n \n RETURN in_new_status = ANY(t_allowed_statuses);\nEND;\n$;\n\nCREATE OR REPLACE FUNCTION private.ord_calculate_total(in_order_id uuid)\nRETURNS numeric\nLANGUAGE sql\nSTABLE\nAS $\n SELECT COALESCE(SUM(quantity * unit_price), 0)\n FROM data.order_items\n WHERE order_id = in_order_id;\n$;\n\n-- ============================================================================\n-- ORDERS MODULE - Public API Functions\n-- ============================================================================\n\nCREATE OR REPLACE FUNCTION api.ord_get(in_id uuid)\nRETURNS TABLE (\n id uuid,\n customer_id uuid,\n customer_email text,\n status text,\n total numeric,\n item_count bigint,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT \n o.id,\n o.customer_id,\n c.email,\n o.status,\n o.total,\n (SELECT COUNT(*) FROM data.order_items WHERE order_id = o.id),\n o.created_at\n FROM data.orders o\n JOIN data.customers c ON c.id = o.customer_id\n WHERE o.id = in_id;\n$;\n\nCREATE OR REPLACE FUNCTION api.ord_select_by_customer(\n in_customer_id uuid,\n in_status text DEFAULT NULL,\n in_limit integer DEFAULT 50\n)\nRETURNS TABLE (\n id uuid,\n status text,\n total numeric,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, status, total, created_at\n FROM data.orders\n WHERE customer_id = in_customer_id\n AND (in_status IS NULL OR status = in_status)\n ORDER BY created_at DESC\n LIMIT in_limit;\n$;\n\n-- ============================================================================\n-- ORDERS MODULE - Public API Procedures\n-- ============================================================================\n\nCREATE OR REPLACE PROCEDURE api.ord_insert(\n in_customer_id uuid,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_customer_active boolean;\nBEGIN\n -- Validate customer exists and is active\n SELECT is_active INTO l_customer_active\n FROM data.customers\n WHERE id = in_customer_id;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Customer not found: %', in_customer_id\n USING ERRCODE = 'P0030';\n END IF;\n \n IF NOT l_customer_active THEN\n RAISE EXCEPTION 'Customer is inactive: %', in_customer_id\n USING ERRCODE = 'P0031';\n END IF;\n \n -- Create order\n INSERT INTO data.orders (customer_id, status, total)\n VALUES (in_customer_id, 'draft', 0)\n RETURNING id INTO io_id;\nEND;\n$;\n\nCREATE OR REPLACE PROCEDURE api.ord_update_status(\n in_id uuid,\n in_new_status text\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_current_status text;\nBEGIN\n -- Get current status\n SELECT status INTO l_current_status\n FROM data.orders\n WHERE id = in_id\n FOR UPDATE;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Order not found: %', in_id\n USING ERRCODE = 'P0020';\n END IF;\n \n -- Validate transition\n IF NOT private.ord_validate_status_transition(l_current_status, in_new_status) THEN\n RAISE EXCEPTION 'Invalid status transition: % -> %', \n l_current_status, in_new_status\n USING ERRCODE = 'P0021';\n END IF;\n \n -- Update status\n UPDATE data.orders\n SET status = in_new_status\n WHERE id = in_id;\n \n -- Recalculate total when confirming\n IF in_new_status = 'confirmed' THEN\n UPDATE data.orders\n SET total = private.ord_calculate_total(in_id)\n WHERE id = in_id;\n END IF;\nEND;\n$;\n\n-- ============================================================================\n-- ORDERS MODULE - Documentation\n-- ============================================================================\n\nCOMMENT ON FUNCTION api.ord_get(uuid) IS \n 'Get order details by ID including customer info and item count';\n\nCOMMENT ON FUNCTION api.ord_select_by_customer(uuid, text, integer) IS \n 'List orders for a customer, optionally filtered by status';\n\nCOMMENT ON PROCEDURE api.ord_insert(uuid, uuid) IS \n 'Create a new draft order for a customer';\n\nCOMMENT ON PROCEDURE api.ord_update_status(uuid, text) IS \n 'Update order status with validation of allowed transitions';\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":37082,"content_sha256":"774c939a6e2966fa943fe2d27636773bb48c5a59a2f267a5ff1606cdd29c992d"},{"filename":"references/data-types.md","content":"# Data Types Best Practices (PostgreSQL 18+)\n\n## Table of Contents\n1. [Primary Keys](#primary-keys)\n2. [Text Types](#text-types)\n3. [Numeric Types](#numeric-types)\n4. [Date/Time Types](#datetime-types)\n5. [Boolean Type](#boolean-type)\n6. [JSON Types](#json-types)\n7. [Array Types](#array-types)\n8. [UUID Types](#uuid-types)\n9. [Types to Avoid](#types-to-avoid)\n\n## Data Type Selection Guide\n\n```mermaid\nflowchart TD\n START([What kind of data?]) --> TYPE{Data Category}\n \n TYPE -->|\"Identifier/PK\"| PK{Needs global uniqueness?}\n TYPE -->|\"Text\"| TXT[\"Use: text\"]\n TYPE -->|\"Money/Precise decimals\"| MONEY[\"Use: numeric(p,s)\"]\n TYPE -->|\"Integers\"| INT{Range needed?}\n TYPE -->|\"Date/Time\"| DT{With timezone?}\n TYPE -->|\"True/False\"| BOOL[\"Use: boolean\"]\n TYPE -->|\"Key-Value/Document\"| JSON[\"Use: jsonb\"]\n TYPE -->|\"List of values\"| ARR[\"Use: array[]\"]\n \n PK -->|\"Yes\"| UUID7[\"Use: uuid DEFAULT uuidv7()\"]\n PK -->|\"No (internal only)\"| IDENTITY[\"Use: GENERATED ALWAYS AS IDENTITY\"]\n \n INT -->|\"\u003c 2 billion\"| INT4[\"Use: integer\"]\n INT -->|\"> 2 billion\"| INT8[\"Use: bigint\"]\n INT -->|\"\u003c 32,000\"| INT2[\"Use: smallint\"]\n \n DT -->|\"Yes (usually!)\"| TSTZ[\"Use: timestamptz\"]\n DT -->|\"No (rare)\"| TS[\"Use: timestamp\"]\n \n style UUID7 fill:#c8e6c9\n style IDENTITY fill:#c8e6c9\n style TXT fill:#c8e6c9\n style MONEY fill:#c8e6c9\n style TSTZ fill:#c8e6c9\n style BOOL fill:#c8e6c9\n style JSON fill:#c8e6c9\n```\n\n### Types to Use vs Avoid\n\n```mermaid\ngraph LR\n subgraph GOOD[\"✅ USE\"]\n G1[\"text\"]\n G2[\"numeric(p,s)\"]\n G3[\"timestamptz\"]\n G4[\"boolean\"]\n G5[\"uuid + uuidv7()\"]\n G6[\"IDENTITY\"]\n G7[\"jsonb\"]\n G8[\"integer/bigint\"]\n end\n \n subgraph BAD[\"❌ AVOID\"]\n B1[\"char(n)\"]\n B2[\"varchar(n)\"]\n B3[\"money\"]\n B4[\"timestamp\"]\n B5[\"serial\"]\n B6[\"json\"]\n B7[\"float/real\"]\n end\n \n B1 -.->|\"use instead\"| G1\n B2 -.->|\"use instead\"| G1\n B3 -.->|\"use instead\"| G2\n B4 -.->|\"use instead\"| G3\n B5 -.->|\"use instead\"| G6\n B6 -.->|\"use instead\"| G7\n B7 -.->|\"use instead\"| G2\n \n style GOOD fill:#c8e6c9\n style BAD fill:#ffcdd2\n```\n\n## Primary Keys\n\n### Recommended: UUIDv7 (PostgreSQL 18+)\n\n```sql\n-- UUIDv7: timestamp-ordered, globally unique, excellent index locality\nCREATE TABLE data.orders (\n id uuid PRIMARY KEY DEFAULT uuidv7()\n);\n\n-- Extract timestamp from UUIDv7\nSELECT uuid_extract_timestamp(id) AS created FROM data.orders;\n```\n\n**Benefits of UUIDv7:**\n- Globally unique without coordination\n- Timestamp-ordered for B-tree index efficiency\n- Better cache locality than random UUIDs\n- Safe for distributed systems\n- No sequence contention\n\n### Alternative: Identity Columns\n\n```sql\n-- For internal tables where global uniqueness not needed\nCREATE TABLE data.audit_log (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY\n);\n\n-- Allow override (rarely needed)\nCREATE TABLE data.imported_data (\n id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY\n);\n```\n\n### Never Use: SERIAL\n\n```sql\n-- DEPRECATED: Don't use serial/bigserial\n-- Bad\nCREATE TABLE bad_example (\n id serial PRIMARY KEY -- Don't do this\n);\n\n-- Good: Use identity instead\nCREATE TABLE good_example (\n id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY\n);\n```\n\n## Text Types\n\n### Use: text\n\n```sql\n-- text: variable unlimited length, best performance\nname text NOT NULL\ndescription text\nemail text NOT NULL\n```\n\n### Use: varchar (without length) for documentation\n\n```sql\n-- varchar without length = same as text, but documents intent\ncountry_code varchar -- Still unlimited, but hints at short values\n```\n\n### Avoid: char(n) and varchar(n)\n\n```sql\n-- AVOID: Fixed-length and length-limited types\n-- These provide no performance benefit and cause issues\n\n-- Bad\ncountry_code char(2) -- Pads with spaces, causes comparison issues\nusername varchar(50) -- Arbitrary limit, may need migration later\n\n-- Good\ncountry_code text CHECK (length(country_code) = 2)\nusername text CHECK (length(username) \u003c= 50) -- If limit truly needed\n```\n\n## Numeric Types\n\n### Integer Types\n\n| Type | Range | Use Case |\n|------|-------|----------|\n| `smallint` | -32,768 to 32,767 | Enum-like values, small counts |\n| `integer` | -2.1B to 2.1B | General purpose, foreign keys |\n| `bigint` | -9.2×10¹⁸ to 9.2×10¹⁸ | Large IDs, timestamps as integers |\n\n```sql\nquantity integer NOT NULL DEFAULT 0\nretry_count smallint NOT NULL DEFAULT 0\ntotal_records bigint NOT NULL DEFAULT 0\n```\n\n### Decimal Types\n\n```sql\n-- numeric(precision, scale): Exact decimal arithmetic\nprice numeric(15, 2) NOT NULL -- Up to 9999999999999.99\ntax_rate numeric(5, 4) NOT NULL -- Up to 9.9999 (99.99%)\nexchange_rate numeric(20, 10) -- High precision rates\n\n-- For monetary values: ALWAYS use numeric, NEVER money type\naccount_balance numeric(19, 4) NOT NULL DEFAULT 0\n```\n\n### Floating Point (Use Sparingly)\n\n```sql\n-- Only for scientific/statistical data where exact precision not critical\nlatitude double precision\nlongitude double precision\nscore real -- 6 decimal digits precision\n```\n\n## Date/Time Types\n\n### Always Use: timestamptz\n\n```sql\n-- timestamptz: timestamp with time zone (stores UTC internally)\ncreated_at timestamptz NOT NULL DEFAULT now()\nupdated_at timestamptz NOT NULL DEFAULT now()\nexpires_at timestamptz\nevent_time timestamptz NOT NULL\n\n-- Date only (no time component)\nbirth_date date\neffective_date date NOT NULL\n\n-- Time only with timezone\nopening_time timetz\n```\n\n### Never Use: timestamp (without time zone)\n\n```sql\n-- NEVER use timestamp without time zone\n-- It loses timezone context and causes bugs\n\n-- Bad\ncreated_at timestamp -- Ambiguous: what timezone?\n\n-- Good\ncreated_at timestamptz -- Always unambiguous\n```\n\n### Intervals\n\n```sql\n-- interval: for durations\nduration interval\nrental_period interval NOT NULL DEFAULT '7 days'\ntimeout interval NOT NULL DEFAULT '30 minutes'\n\n-- Interval arithmetic\nSELECT now() + interval '1 hour';\nSELECT age(now(), created_at);\n```\n\n### Date/Time Ranges (PostgreSQL 18 Temporal Constraints)\n\n```sql\n-- Range types for temporal data\nCREATE TABLE data.reservations (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n room_id uuid NOT NULL,\n guest_id uuid NOT NULL,\n during tstzrange NOT NULL, -- [checkin, checkout)\n \n -- Temporal primary key (PG18)\n CONSTRAINT reservations_no_double_booking \n PRIMARY KEY (room_id, during WITHOUT OVERLAPS)\n);\n\n-- Insert with range\nINSERT INTO data.reservations (room_id, guest_id, during)\nVALUES (\n '...', \n '...', \n tstzrange('2024-03-01 14:00', '2024-03-05 11:00', '[)')\n);\n```\n\n## Boolean Type\n\n```sql\n-- Always use boolean, never integer flags\nis_active boolean NOT NULL DEFAULT true\nis_verified boolean NOT NULL DEFAULT false\nhas_accepted boolean NOT NULL DEFAULT false\nemail_confirmed boolean NOT NULL DEFAULT false\n\n-- With check constraint for documentation\nis_admin boolean NOT NULL DEFAULT false \n CONSTRAINT users_is_admin_boolean CHECK (is_admin IN (true, false))\n```\n\n## JSON Types\n\n### Use: jsonb (Almost Always)\n\n```sql\n-- jsonb: binary format, indexable, faster operations\nmetadata jsonb NOT NULL DEFAULT '{}'\nsettings jsonb NOT NULL DEFAULT '{}'\nattributes jsonb\n\n-- With validation\nCREATE TABLE data.products (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n data jsonb NOT NULL,\n CONSTRAINT products_data_valid CHECK (\n data ? 'name' AND \n data ? 'price' AND \n jsonb_typeof(data->'price') = 'number'\n )\n);\n\n-- GIN index for jsonb queries\nCREATE INDEX products_data_idx ON data.products USING gin (data);\n\n-- Query jsonb\nSELECT * FROM data.products WHERE data @> '{\"category\": \"electronics\"}';\nSELECT * FROM data.products WHERE data->>'name' ILIKE '%widget%';\n```\n\n### Rare: json (Preserve Formatting)\n\n```sql\n-- json: text format, preserves key order and whitespace\n-- Only use when exact preservation required\nraw_api_response json -- Preserve original formatting\n```\n\n## Array Types\n\n```sql\n-- Array columns\ntags text[] NOT NULL DEFAULT '{}'\nscores integer[]\ncoordinates double precision[2]\n\n-- With constraints\nCREATE TABLE data.articles (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n tags text[] NOT NULL DEFAULT '{}',\n CONSTRAINT articles_tags_not_empty CHECK (array_length(tags, 1) > 0)\n);\n\n-- GIN index for array contains queries\nCREATE INDEX articles_tags_idx ON data.articles USING gin (tags);\n\n-- Query arrays\nSELECT * FROM data.articles WHERE tags @> ARRAY['postgresql'];\nSELECT * FROM data.articles WHERE 'postgresql' = ANY(tags);\n```\n\n## UUID Types\n\n### UUIDv7 for Primary Keys (PostgreSQL 18+)\n\n```sql\n-- New in PostgreSQL 18: timestamp-ordered UUIDs\nid uuid PRIMARY KEY DEFAULT uuidv7()\n\n-- Extract timestamp\nSELECT uuid_extract_timestamp(uuidv7()); -- Returns timestamptz\n```\n\n### UUIDv4 for Random Identifiers\n\n```sql\n-- Random UUID (use for tokens, not primary keys)\napi_token uuid DEFAULT gen_random_uuid()\n\n-- Note: uuidv4() is now alias for gen_random_uuid() in PG18\n```\n\n## Hierarchical Data with ltree\n\n### Install ltree Extension\n\n```sql\nCREATE EXTENSION IF NOT EXISTS ltree;\n```\n\n### ltree Data Type\n\n```sql\n-- ltree stores hierarchical paths like: 'Root.Science.Biology.Genetics'\n-- Labels separated by dots, each label is alphanumeric\n\nCREATE TABLE data.categories (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n path ltree NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Create indexes for efficient queries\nCREATE INDEX categories_path_gist_idx ON data.categories USING gist (path);\nCREATE INDEX categories_path_btree_idx ON data.categories USING btree (path);\n\n-- Insert hierarchical data\nINSERT INTO data.categories (name, path) VALUES\n ('Products', 'Products'),\n ('Electronics', 'Products.Electronics'),\n ('Phones', 'Products.Electronics.Phones'),\n ('Computers', 'Products.Electronics.Computers'),\n ('Laptops', 'Products.Electronics.Computers.Laptops'),\n ('Clothing', 'Products.Clothing'),\n ('Shirts', 'Products.Clothing.Shirts');\n```\n\n### ltree Queries\n\n```sql\n-- Find all descendants of Electronics\nSELECT * FROM data.categories\nWHERE path \u003c@ 'Products.Electronics';\n-- Returns: Electronics, Phones, Computers, Laptops\n\n-- Find all ancestors of Laptops\nSELECT * FROM data.categories\nWHERE path @> 'Products.Electronics.Computers.Laptops';\n-- Returns: Products, Electronics, Computers, Laptops\n\n-- Find direct children\nSELECT * FROM data.categories\nWHERE path ~ 'Products.Electronics.*{1}';\n-- Returns: Phones, Computers (one level deep)\n\n-- Find items at specific depth\nSELECT * FROM data.categories\nWHERE nlevel(path) = 3;\n-- Returns items at depth 3\n\n-- Get parent path\nSELECT name, subpath(path, 0, -1) AS parent_path\nFROM data.categories\nWHERE name = 'Laptops';\n-- Returns: Products.Electronics.Computers\n```\n\n### ltree Operators\n\n```sql\n-- @> ancestor of (or equal)\n-- \u003c@ descendant of (or equal)\n-- ~ match lquery pattern\n-- ? match any ltxtquery\n-- || concatenate paths\n\n-- Pattern matching with lquery\nSELECT * FROM data.categories\nWHERE path ~ 'Products.*.Computers.*'; -- Any path containing Computers\n\n-- Text search with ltxtquery\nSELECT * FROM data.categories\nWHERE path ? 'Elect* & !Phones'; -- Contains Elect*, not Phones\n```\n\n### Building Hierarchical Trees\n\n```sql\n-- Get full category tree with depth\nCREATE FUNCTION api.get_category_tree(in_root_path ltree DEFAULT 'Products')\nRETURNS TABLE (\n id uuid,\n name text,\n path ltree,\n depth integer\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, name, path, nlevel(path) - nlevel(in_root_path) AS depth\n FROM data.categories\n WHERE path \u003c@ in_root_path\n ORDER BY path;\n$;\n\n-- Move subtree\nCREATE PROCEDURE api.move_category(\n in_category_id uuid,\n in_new_parent_path ltree\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_old_path ltree;\n l_new_path ltree;\n l_category_name text;\nBEGIN\n SELECT path, name INTO l_old_path, l_category_name\n FROM data.categories WHERE id = in_category_id;\n\n l_new_path := in_new_parent_path || l_category_name;\n\n -- Update category and all descendants\n UPDATE data.categories\n SET path = l_new_path || subpath(path, nlevel(l_old_path))\n WHERE path \u003c@ l_old_path;\nEND;\n$;\n```\n\n## Types to Avoid\n\n### Never Use These Types\n\n| Type | Problem | Use Instead |\n|------|---------|-------------|\n| `char(n)` | Space padding, comparison issues | `text` |\n| `varchar(n)` | Arbitrary limits, no performance benefit | `text` with CHECK |\n| `money` | Locale-dependent, rounding issues | `numeric(19,4)` |\n| `serial` | Legacy, permission issues | `GENERATED AS IDENTITY` |\n| `bigserial` | Legacy, permission issues | `bigint GENERATED AS IDENTITY` |\n| `timestamp` | No timezone, ambiguous | `timestamptz` |\n| `timetz` | Rarely useful, confusing semantics | Reconsider design |\n\n### Type Migration Examples\n\n```sql\n-- Migrating from serial to identity\nALTER TABLE data.old_table \n ALTER COLUMN id DROP DEFAULT,\n ALTER COLUMN id ADD GENERATED ALWAYS AS IDENTITY;\n\n-- Migrating from timestamp to timestamptz\nALTER TABLE data.events \n ALTER COLUMN event_time TYPE timestamptz \n USING event_time AT TIME ZONE 'UTC';\n\n-- Migrating from money to numeric\nALTER TABLE data.products\n ALTER COLUMN price TYPE numeric(15,2) \n USING price::numeric;\n```\n\n## Generated Columns (PostgreSQL 18+)\n\n### Virtual Generated Columns (Default in PG18)\n\n```sql\n-- Computed at query time, no storage\nCREATE TABLE data.orders (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n subtotal numeric(15,2) NOT NULL,\n tax_rate numeric(5,4) NOT NULL DEFAULT 0.0875,\n -- Virtual (computed when read)\n total numeric(15,2) GENERATED ALWAYS AS (subtotal * (1 + tax_rate))\n);\n```\n\n### Stored Generated Columns\n\n```sql\n-- Computed and stored on write (use when computation expensive)\nCREATE TABLE data.users (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n first_name text NOT NULL,\n last_name text NOT NULL,\n -- Stored (can be indexed, replicated)\n full_name text GENERATED ALWAYS AS (first_name || ' ' || last_name) STORED\n);\n\n-- Can index stored generated columns\nCREATE INDEX users_full_name_idx ON data.users(full_name);\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":14480,"content_sha256":"51fc2ca1d050dcf9738a6a9f2f2da23972a9eeda02fe94966ba7a946ffdc9dbb"},{"filename":"references/data-warehousing-medallion.md","content":"# Data Warehousing with Medallion Architecture\n\nA comprehensive guide to implementing a modular, maintainable, and performant data warehouse in PostgreSQL using the Medallion Architecture pattern, integrated with this skill's schema separation and Table API conventions.\n\n## Table of Contents\n\n1. [Medallion Architecture Overview](#medallion-architecture-overview)\n2. [Schema Organization](#schema-organization)\n3. [Bronze Layer (Raw)](#bronze-layer-raw)\n4. [Silver Layer (Cleansed)](#silver-layer-cleansed)\n5. [Gold Layer (Business)](#gold-layer-business)\n6. [Data Lineage Tracking](#data-lineage-tracking)\n7. [ETL Orchestration](#etl-orchestration)\n8. [Incremental Processing](#incremental-processing)\n9. [Data Quality Framework](#data-quality-framework)\n10. [Performance Optimization](#performance-optimization)\n11. [Complete Implementation Example](#complete-implementation-example)\n\n---\n\n## Medallion Architecture Overview\n\nThe Medallion Architecture organizes data into three progressive layers of refinement, each serving a specific purpose in the data pipeline.\n\n### Architecture Diagram\n\n```mermaid\nflowchart TB\n subgraph SOURCES[\"📥 Source Systems\"]\n direction LR\n S1[(\"OLTP\u003cbr/>Database\")]\n S2[(\"APIs\")]\n S3[(\"Files\u003cbr/>CSV/JSON\")]\n S4[(\"Events\u003cbr/>Kafka/SQS\")]\n end\n \n subgraph BRONZE[\"🥉 Bronze Layer (Raw)\"]\n direction LR\n B1[(\"raw_customers\")]\n B2[(\"raw_orders\")]\n B3[(\"raw_events\")]\n B_DESC[\"• Exact copy of source\u003cbr/>• Append-only\u003cbr/>• Full history\u003cbr/>• No transformations\"]\n end\n \n subgraph SILVER[\"🥈 Silver Layer (Cleansed)\"]\n direction LR\n SV1[(\"customers\")]\n SV2[(\"orders\")]\n SV3[(\"products\")]\n S_DESC[\"• Deduplicated\u003cbr/>• Validated\u003cbr/>• Type-cast\u003cbr/>• SCD Type 2\"]\n end\n \n subgraph GOLD[\"🥇 Gold Layer (Business)\"]\n direction LR\n G1[(\"dim_customer\")]\n G2[(\"fact_sales\")]\n G3[(\"agg_daily_sales\")]\n G_DESC[\"• Star schema\u003cbr/>• Aggregates\u003cbr/>• KPIs\u003cbr/>• BI-ready\"]\n end\n \n subgraph LINEAGE[\"📊 Lineage Schema\"]\n L1[(\"pipeline_runs\")]\n L2[(\"table_lineage\")]\n L3[(\"column_lineage\")]\n end\n \n SOURCES --> BRONZE\n BRONZE --> SILVER\n SILVER --> GOLD\n \n BRONZE -.->|\"tracks\"| LINEAGE\n SILVER -.->|\"tracks\"| LINEAGE\n GOLD -.->|\"tracks\"| LINEAGE\n \n style BRONZE fill:#cd7f32,color:#fff\n style SILVER fill:#c0c0c0,color:#000\n style GOLD fill:#ffd700,color:#000\n style LINEAGE fill:#e1f5fe\n```\n\n### Layer Responsibilities\n\n| Layer | Purpose | Data Characteristics | Retention |\n|-------|---------|---------------------|-----------|\n| **Bronze** | Landing zone | Raw, unmodified, append-only | Long (years) |\n| **Silver** | Single source of truth | Cleansed, validated, deduplicated | Medium (months) |\n| **Gold** | Business consumption | Aggregated, denormalized, optimized | Configurable |\n\n### Key Principles\n\n1. **Immutability**: Bronze data is never modified after landing\n2. **Traceability**: Every record can be traced back to its source\n3. **Replayability**: Any layer can be rebuilt from the layer below\n4. **Modularity**: Each layer is independent and testable\n5. **Progressive Refinement**: Data quality improves through each layer\n\n---\n\n## Schema Organization\n\n### Recommended Schema Structure\n\nI recommend a **separate schema per layer** with a dedicated **lineage schema** for tracking:\n\n```mermaid\nflowchart LR\n subgraph DWH[\"Data Warehouse Schemas\"]\n direction TB\n \n subgraph MEDAL[\"Medallion Layers\"]\n BRONZE_S[\"bronze\u003cbr/>───────\u003cbr/>Raw tables\u003cbr/>Landing zone\"]\n SILVER_S[\"silver\u003cbr/>───────\u003cbr/>Cleansed tables\u003cbr/>Conformed dims\"]\n GOLD_S[\"gold\u003cbr/>───────\u003cbr/>Star schemas\u003cbr/>Aggregates\"]\n end\n \n subgraph SUPPORT[\"Support Schemas\"]\n LINEAGE_S[\"dwh_lineage\u003cbr/>───────\u003cbr/>Pipeline tracking\u003cbr/>Data lineage\"]\n ETL_S[\"dwh_etl\u003cbr/>───────\u003cbr/>Transform functions\u003cbr/>Load procedures\"]\n STAGING_S[\"dwh_staging\u003cbr/>───────\u003cbr/>Temp processing\u003cbr/>Work tables\"]\n end\n end\n \n BRONZE_S --> SILVER_S --> GOLD_S\n ETL_S -->|\"transforms\"| MEDAL\n MEDAL -->|\"logs to\"| LINEAGE_S\n \n style BRONZE_S fill:#cd7f32,color:#fff\n style SILVER_S fill:#c0c0c0\n style GOLD_S fill:#ffd700\n style LINEAGE_S fill:#e3f2fd\n style ETL_S fill:#fff3e0\n style STAGING_S fill:#f3e5f5\n```\n\n### Why Separate Lineage Schema?\n\nI recommend a dedicated `dwh_lineage` schema for data lineage because:\n\n1. **Cross-cutting concern**: Lineage spans all medallion layers and shouldn't belong to any single one\n2. **Different access patterns**: Data engineers need lineage access; analysts typically don't\n3. **Independent lifecycle**: Lineage can be upgraded/modified without touching data schemas\n4. **Clear separation**: Keeps operational metadata distinct from business data\n5. **Compliance**: Makes it easy to demonstrate data provenance for audits\n\n### Schema Creation Script\n\n```sql\n-- ============================================================================\n-- Data Warehouse Schema Setup\n-- ============================================================================\n\n-- Medallion layers\nCREATE SCHEMA IF NOT EXISTS bronze;\nCOMMENT ON SCHEMA bronze IS 'Raw data landing zone - exact copies from sources';\n\nCREATE SCHEMA IF NOT EXISTS silver;\nCOMMENT ON SCHEMA silver IS 'Cleansed and validated data - single source of truth';\n\nCREATE SCHEMA IF NOT EXISTS gold;\nCOMMENT ON SCHEMA gold IS 'Business-ready data - star schemas and aggregates';\n\n-- Support schemas\nCREATE SCHEMA IF NOT EXISTS dwh_lineage;\nCOMMENT ON SCHEMA dwh_lineage IS 'Data lineage tracking and pipeline metadata';\n\nCREATE SCHEMA IF NOT EXISTS dwh_etl;\nCOMMENT ON SCHEMA dwh_etl IS 'ETL functions, procedures, and orchestration';\n\nCREATE SCHEMA IF NOT EXISTS dwh_staging;\nCOMMENT ON SCHEMA dwh_staging IS 'Temporary staging tables for ETL processing';\n\n-- Revoke public access\nREVOKE ALL ON SCHEMA bronze FROM PUBLIC;\nREVOKE ALL ON SCHEMA silver FROM PUBLIC;\nREVOKE ALL ON SCHEMA gold FROM PUBLIC;\nREVOKE ALL ON SCHEMA dwh_lineage FROM PUBLIC;\nREVOKE ALL ON SCHEMA dwh_etl FROM PUBLIC;\nREVOKE ALL ON SCHEMA dwh_staging FROM PUBLIC;\n```\n\n### ETL Security Model\n\nUnlike OLTP applications (which use `SECURITY DEFINER` functions to access data through the `api` schema), ETL pipelines typically run under a **dedicated service account** with direct schema access:\n\n```sql\n-- Create ETL service role\nCREATE ROLE etl_service LOGIN PASSWORD 'secure_password';\n\n-- Grant access to all DWH schemas\nGRANT USAGE ON SCHEMA bronze, silver, gold, dwh_lineage, dwh_etl, dwh_staging TO etl_service;\nGRANT ALL ON ALL TABLES IN SCHEMA bronze, silver, gold, dwh_staging TO etl_service;\nGRANT SELECT, INSERT ON ALL TABLES IN SCHEMA dwh_lineage TO etl_service;\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA dwh_etl TO etl_service;\nGRANT EXECUTE ON ALL PROCEDURES IN SCHEMA dwh_etl TO etl_service;\n\n-- Default privileges for future objects\nALTER DEFAULT PRIVILEGES IN SCHEMA bronze, silver, gold \n GRANT ALL ON TABLES TO etl_service;\n```\n\n**Why no SECURITY DEFINER for ETL?**\n- ETL runs as a scheduled job, not user-initiated requests\n- The service account is already trusted with full DWH access \n- Simpler debugging (no context switching)\n- Better performance (no overhead from SECURITY DEFINER)\n\n**For analyst/BI access** to the gold layer, create a separate read-only role:\n\n```sql\nCREATE ROLE analyst_role;\nGRANT USAGE ON SCHEMA gold TO analyst_role;\nGRANT SELECT ON ALL TABLES IN SCHEMA gold TO analyst_role;\nALTER DEFAULT PRIVILEGES IN SCHEMA gold GRANT SELECT ON TABLES TO analyst_role;\n```\n\n### Integration with OLTP Schemas\n\nIf your database has both OLTP (`data`, `api`, `private`) and DWH schemas:\n\n```mermaid\nflowchart TB\n subgraph OLTP[\"OLTP Schemas (Existing Skill)\"]\n API[\"api\"]\n DATA[\"data\"]\n PRIV[\"private\"]\n end\n \n subgraph DWH[\"Data Warehouse Schemas\"]\n BRONZE[\"bronze\"]\n SILVER[\"silver\"]\n GOLD[\"gold\"]\n LINEAGE[\"dwh_lineage\"]\n ETL[\"dwh_etl\"]\n end\n \n DATA -->|\"CDC/Extract\"| BRONZE\n BRONZE --> SILVER --> GOLD\n \n ETL -->|\"transforms\"| DWH\n DWH -->|\"logs\"| LINEAGE\n \n style OLTP fill:#e8f5e9\n style DWH fill:#fff8e1\n```\n\n---\n\n## Bronze Layer (Raw)\n\nThe bronze layer is the **landing zone** for all source data. Data here should be:\n- **Exact copies** of source systems\n- **Append-only** (never updated or deleted)\n- **Timestamped** with ingestion metadata\n- **Unmodified** from source format\n\n### Bronze Table Design Pattern\n\n```sql\n-- ============================================================================\n-- Bronze Table Template\n-- ============================================================================\n\nCREATE TABLE bronze.raw_customers (\n -- Ingestion metadata (always first)\n _bronze_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n _ingested_at timestamptz NOT NULL DEFAULT now(),\n _source_system text NOT NULL,\n _source_file text, -- For file-based sources\n _batch_id uuid, -- Links to pipeline run\n _raw_payload jsonb, -- Optional: store original JSON\n \n -- Source data (exact copy, nullable to handle bad data)\n id text, -- Keep as text initially\n email text,\n name text,\n status text,\n created_date text, -- Don't parse yet\n updated_date text,\n extra_field_1 text, -- Unknown/extra fields\n extra_field_2 text\n);\n\n-- Partition by ingestion date for efficient pruning\nCREATE TABLE bronze.raw_customers_partitioned (\n _bronze_id bigint GENERATED ALWAYS AS IDENTITY,\n _ingested_at timestamptz NOT NULL DEFAULT now(),\n _source_system text NOT NULL,\n _batch_id uuid,\n \n -- Source columns\n id text,\n email text,\n name text,\n created_date text,\n \n PRIMARY KEY (_bronze_id, _ingested_at)\n) PARTITION BY RANGE (_ingested_at);\n\n-- Create monthly partitions\nCREATE TABLE bronze.raw_customers_2024_01 \n PARTITION OF bronze.raw_customers_partitioned\n FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');\n```\n\n### Bronze Ingestion Patterns\n\n#### Pattern 1: Batch File Ingestion\n\n```sql\nCREATE OR REPLACE PROCEDURE dwh_etl.ingest_customers_csv(\n in_file_path text,\n in_source text DEFAULT 'csv_import',\n INOUT io_batch_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_row_count integer;\nBEGIN\n -- Generate batch ID\n io_batch_id := COALESCE(io_batch_id, gen_random_uuid());\n \n -- Create temp table for COPY\n CREATE TEMP TABLE temp_csv_import (\n id text,\n email text,\n name text,\n status text,\n created_date text,\n updated_date text\n ) ON COMMIT DROP;\n \n -- Load CSV\n EXECUTE format('COPY temp_csv_import FROM %L WITH CSV HEADER', in_file_path);\n \n -- Insert into bronze with metadata\n INSERT INTO bronze.raw_customers (\n _source_system, _source_file, _batch_id,\n id, email, name, status, created_date, updated_date\n )\n SELECT \n in_source,\n in_file_path,\n io_batch_id,\n id, email, name, status, created_date, updated_date\n FROM temp_csv_import;\n \n GET DIAGNOSTICS l_row_count = ROW_COUNT;\n \n -- Log to lineage\n PERFORM dwh_lineage.log_ingestion(\n in_table_name := 'bronze.raw_customers',\n in_batch_id := io_batch_id,\n in_source := in_source,\n in_row_count := l_row_count\n );\n \n RAISE NOTICE 'Ingested % rows into bronze.raw_customers (batch: %)', \n l_row_count, io_batch_id;\nEND;\n$;\n```\n\n#### Pattern 2: CDC (Change Data Capture) Ingestion\n\n```sql\n-- Bronze table for CDC events\nCREATE TABLE bronze.raw_customers_cdc (\n _bronze_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n _ingested_at timestamptz NOT NULL DEFAULT now(),\n _cdc_operation text NOT NULL, -- INSERT, UPDATE, DELETE\n _cdc_timestamp timestamptz NOT NULL, -- When change occurred at source\n _cdc_lsn text, -- Log sequence number\n _batch_id uuid,\n \n -- Full row data (before/after)\n before_data jsonb, -- Previous values (UPDATE/DELETE)\n after_data jsonb -- New values (INSERT/UPDATE)\n);\n\n-- Ingest CDC events\nCREATE OR REPLACE PROCEDURE dwh_etl.ingest_cdc_event(\n in_operation text,\n in_timestamp timestamptz,\n in_before jsonb,\n in_after jsonb,\n in_batch_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nAS $\nBEGIN\n INSERT INTO bronze.raw_customers_cdc (\n _cdc_operation, _cdc_timestamp, _batch_id,\n before_data, after_data\n ) VALUES (\n in_operation, in_timestamp, COALESCE(in_batch_id, gen_random_uuid()),\n in_before, in_after\n );\nEND;\n$;\n```\n\n#### Pattern 3: API/JSON Ingestion\n\n```sql\nCREATE OR REPLACE PROCEDURE dwh_etl.ingest_api_response(\n in_endpoint text,\n in_response jsonb,\n INOUT io_batch_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_row_count integer;\nBEGIN\n io_batch_id := COALESCE(io_batch_id, gen_random_uuid());\n \n -- Store raw JSON array elements\n INSERT INTO bronze.raw_api_responses (\n _source_system, _batch_id, _raw_payload,\n id, email, name\n )\n SELECT \n in_endpoint,\n io_batch_id,\n item,\n item->>'id',\n item->>'email',\n item->>'name'\n FROM jsonb_array_elements(in_response->'data') AS item;\n \n GET DIAGNOSTICS l_row_count = ROW_COUNT;\n \n PERFORM dwh_lineage.log_ingestion(\n in_table_name := 'bronze.raw_api_responses',\n in_batch_id := io_batch_id,\n in_source := in_endpoint,\n in_row_count := l_row_count\n );\nEND;\n$;\n```\n\n### Bronze Best Practices\n\n| Practice | Rationale |\n|----------|-----------|\n| Never modify bronze data | Enables replay and audit |\n| Use text types initially | Avoids parse errors on bad data |\n| Include ingestion metadata | Enables debugging and lineage |\n| Partition by ingestion date | Efficient pruning and archival |\n| Store raw JSON when possible | Preserves original structure |\n| Use append-only pattern | Simplifies concurrency |\n\n---\n\n## Silver Layer (Cleansed)\n\nThe silver layer transforms bronze data into a **cleansed, validated, and conformed** state. This is your **single source of truth**.\n\n### Silver Layer Characteristics\n\n```mermaid\nflowchart LR\n subgraph BRONZE[\"Bronze Input\"]\n B1[\"Duplicates\"]\n B2[\"Bad formats\"]\n B3[\"Missing values\"]\n B4[\"Inconsistent types\"]\n end\n \n subgraph TRANSFORM[\"Silver Transformations\"]\n T1[\"Deduplicate\"]\n T2[\"Parse & validate\"]\n T3[\"Apply defaults\"]\n T4[\"Type casting\"]\n T5[\"SCD Type 2\"]\n end\n \n subgraph SILVER[\"Silver Output\"]\n S1[\"Clean records\"]\n S2[\"Proper types\"]\n S3[\"Valid constraints\"]\n S4[\"History tracking\"]\n end\n \n B1 --> T1 --> S1\n B2 --> T2 --> S2\n B3 --> T3 --> S3\n B4 --> T4 --> S4\n \n style BRONZE fill:#cd7f32,color:#fff\n style SILVER fill:#c0c0c0\n```\n\n### Silver Table Design Pattern\n\n```sql\n-- ============================================================================\n-- Silver Table with SCD Type 2 (Slowly Changing Dimension)\n-- ============================================================================\n\nCREATE TABLE silver.customers (\n -- Surrogate key\n customer_sk bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n \n -- Natural/business key\n customer_id uuid NOT NULL, -- Parsed from bronze\n \n -- Cleansed attributes\n email text NOT NULL,\n email_domain text GENERATED ALWAYS AS (split_part(email, '@', 2)) STORED,\n name text NOT NULL,\n status text NOT NULL CHECK (status IN ('active', 'inactive', 'suspended')),\n \n -- SCD Type 2 tracking\n valid_from timestamptz NOT NULL DEFAULT now(),\n valid_to timestamptz, -- NULL = current record\n is_current boolean NOT NULL DEFAULT true,\n \n -- Lineage\n _source_bronze_id bigint, -- Links to bronze record\n _batch_id uuid NOT NULL,\n _loaded_at timestamptz NOT NULL DEFAULT now(),\n \n -- Constraints\n CONSTRAINT customers_valid_range CHECK (valid_to IS NULL OR valid_to > valid_from)\n);\n\n-- Unique constraint on business key for current records\nCREATE UNIQUE INDEX silver_customers_current_key \n ON silver.customers(customer_id) \n WHERE is_current = true;\n\n-- Index for SCD lookups\nCREATE INDEX silver_customers_history_idx \n ON silver.customers(customer_id, valid_from, valid_to);\n```\n\n### SCD Type 2 Implementation\n\n```sql\n-- ============================================================================\n-- SCD Type 2 Merge Pattern\n-- ============================================================================\n\nCREATE OR REPLACE PROCEDURE dwh_etl.load_silver_customers(\n in_batch_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_batch_id uuid;\n l_inserted integer := 0;\n l_updated integer := 0;\n l_unchanged integer := 0;\nBEGIN\n l_batch_id := COALESCE(in_batch_id, gen_random_uuid());\n \n -- Step 1: Create staging table with cleansed data\n CREATE TEMP TABLE stg_customers AS\n SELECT DISTINCT ON (id)\n -- Parse and cleanse\n id::uuid AS customer_id,\n lower(trim(email)) AS email,\n trim(name) AS name,\n CASE lower(trim(status))\n WHEN 'active' THEN 'active'\n WHEN 'inactive' THEN 'inactive'\n WHEN 'suspended' THEN 'suspended'\n ELSE 'inactive' -- Default for unknown\n END AS status,\n _bronze_id,\n -- Hash for change detection\n md5(lower(trim(email)) || trim(name) || lower(trim(status))) AS row_hash\n FROM bronze.raw_customers\n WHERE _ingested_at > COALESCE(\n (SELECT MAX(_loaded_at) FROM silver.customers), \n '1900-01-01'::timestamptz\n )\n ORDER BY id, _ingested_at DESC; -- Latest record per ID\n \n -- Step 2: Identify changes\n CREATE TEMP TABLE changes AS\n SELECT \n stg.*,\n cur.customer_sk AS existing_sk,\n CASE \n WHEN cur.customer_sk IS NULL THEN 'INSERT'\n WHEN md5(cur.email || cur.name || cur.status) != stg.row_hash THEN 'UPDATE'\n ELSE 'UNCHANGED'\n END AS change_type\n FROM stg_customers stg\n LEFT JOIN silver.customers cur \n ON cur.customer_id = stg.customer_id \n AND cur.is_current = true;\n \n -- Step 3: Close existing records for updates (SCD Type 2)\n UPDATE silver.customers c\n SET valid_to = now(),\n is_current = false\n FROM changes ch\n WHERE c.customer_sk = ch.existing_sk\n AND ch.change_type = 'UPDATE';\n \n GET DIAGNOSTICS l_updated = ROW_COUNT;\n \n -- Step 4: Insert new and updated records\n INSERT INTO silver.customers (\n customer_id, email, name, status,\n valid_from, is_current,\n _source_bronze_id, _batch_id\n )\n SELECT \n customer_id, email, name, status,\n now(), true,\n _bronze_id, l_batch_id\n FROM changes\n WHERE change_type IN ('INSERT', 'UPDATE');\n \n GET DIAGNOSTICS l_inserted = ROW_COUNT;\n \n SELECT COUNT(*) INTO l_unchanged FROM changes WHERE change_type = 'UNCHANGED';\n \n -- Log to lineage\n PERFORM dwh_lineage.log_transformation(\n in_source_table := 'bronze.raw_customers',\n in_target_table := 'silver.customers',\n in_batch_id := l_batch_id,\n in_rows_inserted := l_inserted,\n in_rows_updated := l_updated,\n in_rows_unchanged := l_unchanged\n );\n \n -- Cleanup\n DROP TABLE stg_customers;\n DROP TABLE changes;\n \n RAISE NOTICE 'Silver customers: % inserted, % updated, % unchanged', \n l_inserted, l_updated, l_unchanged;\nEND;\n$;\n```\n\n### Silver Data Quality Checks\n\n```sql\n-- Apply quality checks during silver load\nCREATE OR REPLACE FUNCTION dwh_etl.validate_customer(\n in_email text,\n in_name text,\n in_status text\n)\nRETURNS TABLE (\n is_valid boolean,\n issues text[]\n)\nLANGUAGE plpgsql\nIMMUTABLE\nAS $\nDECLARE\n l_issues text[] := '{}';\nBEGIN\n -- Email validation\n IF in_email IS NULL OR in_email = '' THEN\n l_issues := array_append(l_issues, 'Email is required');\n ELSIF in_email !~ '^[^@]+@[^@]+\\.[^@]+

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

THEN\n l_issues := array_append(l_issues, 'Invalid email format');\n END IF;\n \n -- Name validation\n IF in_name IS NULL OR length(trim(in_name)) \u003c 2 THEN\n l_issues := array_append(l_issues, 'Name must be at least 2 characters');\n END IF;\n \n -- Status validation\n IF in_status NOT IN ('active', 'inactive', 'suspended') THEN\n l_issues := array_append(l_issues, 'Invalid status value');\n END IF;\n \n RETURN QUERY SELECT array_length(l_issues, 1) = 0 OR l_issues = '{}', l_issues;\nEND;\n$;\n\n-- Log invalid records for review\nCREATE TABLE silver.quality_exceptions (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n source_table text NOT NULL,\n source_id text,\n bronze_id bigint,\n issues text[] NOT NULL,\n source_data jsonb,\n detected_at timestamptz NOT NULL DEFAULT now(),\n resolved_at timestamptz,\n resolution text\n);\n```\n\n---\n\n## Gold Layer (Business)\n\nThe gold layer provides **business-ready data** optimized for consumption by BI tools, reports, and analytics.\n\n### Gold Layer Structure\n\n```mermaid\nflowchart TB\n subgraph SILVER[\"Silver Tables\"]\n S1[\"silver.customers\"]\n S2[\"silver.orders\"]\n S3[\"silver.products\"]\n S4[\"silver.order_items\"]\n end\n \n subgraph GOLD[\"Gold Layer\"]\n subgraph DIMS[\"Dimension Tables\"]\n D1[\"dim_customer\"]\n D2[\"dim_product\"]\n D3[\"dim_date\"]\n D4[\"dim_geography\"]\n end\n \n subgraph FACTS[\"Fact Tables\"]\n F1[\"fact_sales\"]\n F2[\"fact_inventory\"]\n end\n \n subgraph AGGS[\"Aggregates\"]\n A1[\"agg_daily_sales\"]\n A2[\"agg_customer_ltv\"]\n A3[\"agg_product_performance\"]\n end\n end\n \n S1 --> D1\n S3 --> D2\n S2 --> F1\n S4 --> F1\n F1 --> A1\n F1 --> A2\n D2 --> A3\n \n style DIMS fill:#e3f2fd\n style FACTS fill:#fff3e0\n style AGGS fill:#e8f5e9\n```\n\n### Dimension Table Pattern\n\n```sql\n-- ============================================================================\n-- Gold Dimension: Customer\n-- ============================================================================\n\nCREATE TABLE gold.dim_customer (\n -- Surrogate key\n customer_key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n \n -- Natural key\n customer_id uuid NOT NULL,\n \n -- Dimension attributes (denormalized for query performance)\n email text NOT NULL,\n email_domain text NOT NULL,\n name text NOT NULL,\n first_name text GENERATED ALWAYS AS (split_part(name, ' ', 1)) STORED,\n last_name text GENERATED ALWAYS AS (\n CASE WHEN position(' ' in name) > 0 \n THEN substring(name from position(' ' in name) + 1)\n ELSE '' \n END\n ) STORED,\n status text NOT NULL,\n \n -- Derived attributes for analysis\n customer_segment text, -- 'high_value', 'regular', 'new'\n acquisition_date date,\n acquisition_channel text,\n \n -- SCD Type 2\n valid_from timestamptz NOT NULL,\n valid_to timestamptz,\n is_current boolean NOT NULL DEFAULT true,\n \n -- Lineage\n _silver_sk bigint, -- Links to silver record\n _batch_id uuid NOT NULL,\n _loaded_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Business key index\nCREATE UNIQUE INDEX gold_dim_customer_bk ON gold.dim_customer(customer_id) WHERE is_current;\n\n-- Common query patterns\nCREATE INDEX dim_customer_segment_idx ON gold.dim_customer(customer_segment) WHERE is_current;\nCREATE INDEX dim_customer_domain_idx ON gold.dim_customer(email_domain) WHERE is_current;\n```\n\n### Date Dimension\n\n```sql\n-- ============================================================================\n-- Gold Dimension: Date (Generated)\n-- ============================================================================\n\nCREATE TABLE gold.dim_date (\n date_key integer PRIMARY KEY, -- YYYYMMDD format\n full_date date NOT NULL UNIQUE,\n \n -- Date parts\n year smallint NOT NULL,\n quarter smallint NOT NULL,\n month smallint NOT NULL,\n week smallint NOT NULL,\n day_of_month smallint NOT NULL,\n day_of_week smallint NOT NULL,\n day_of_year smallint NOT NULL,\n \n -- Names\n month_name text NOT NULL,\n month_short text NOT NULL,\n day_name text NOT NULL,\n day_short text NOT NULL,\n quarter_name text NOT NULL,\n \n -- Flags\n is_weekend boolean NOT NULL,\n is_holiday boolean NOT NULL DEFAULT false,\n holiday_name text,\n \n -- Fiscal calendar (adjust as needed)\n fiscal_year smallint NOT NULL,\n fiscal_quarter smallint NOT NULL,\n fiscal_month smallint NOT NULL,\n \n -- Relative flags (updated periodically)\n is_current_day boolean NOT NULL DEFAULT false,\n is_current_week boolean NOT NULL DEFAULT false,\n is_current_month boolean NOT NULL DEFAULT false,\n is_current_quarter boolean NOT NULL DEFAULT false,\n is_current_year boolean NOT NULL DEFAULT false\n);\n\n-- Generate date dimension\nCREATE OR REPLACE PROCEDURE dwh_etl.generate_dim_date(\n in_start_date date DEFAULT '2020-01-01',\n in_end_date date DEFAULT '2030-12-31'\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_date date;\nBEGIN\n l_date := in_start_date;\n \n WHILE l_date \u003c= in_end_date LOOP\n INSERT INTO gold.dim_date (\n date_key, full_date,\n year, quarter, month, week, day_of_month, day_of_week, day_of_year,\n month_name, month_short, day_name, day_short, quarter_name,\n is_weekend,\n fiscal_year, fiscal_quarter, fiscal_month\n ) VALUES (\n to_char(l_date, 'YYYYMMDD')::integer,\n l_date,\n EXTRACT(year FROM l_date),\n EXTRACT(quarter FROM l_date),\n EXTRACT(month FROM l_date),\n EXTRACT(week FROM l_date),\n EXTRACT(day FROM l_date),\n EXTRACT(dow FROM l_date),\n EXTRACT(doy FROM l_date),\n to_char(l_date, 'Month'),\n to_char(l_date, 'Mon'),\n to_char(l_date, 'Day'),\n to_char(l_date, 'Dy'),\n 'Q' || EXTRACT(quarter FROM l_date),\n EXTRACT(dow FROM l_date) IN (0, 6),\n -- Fiscal year starting July (adjust as needed)\n CASE WHEN EXTRACT(month FROM l_date) >= 7 \n THEN EXTRACT(year FROM l_date) + 1 \n ELSE EXTRACT(year FROM l_date) END,\n CASE WHEN EXTRACT(month FROM l_date) >= 7 \n THEN EXTRACT(quarter FROM l_date) - 2 \n ELSE EXTRACT(quarter FROM l_date) + 2 END,\n CASE WHEN EXTRACT(month FROM l_date) >= 7 \n THEN EXTRACT(month FROM l_date) - 6 \n ELSE EXTRACT(month FROM l_date) + 6 END\n )\n ON CONFLICT (date_key) DO NOTHING;\n \n l_date := l_date + 1;\n END LOOP;\nEND;\n$;\n\n-- Update relative flags daily\nCREATE OR REPLACE PROCEDURE dwh_etl.update_dim_date_flags()\nLANGUAGE plpgsql\nAS $\nBEGIN\n UPDATE gold.dim_date\n SET is_current_day = (full_date = CURRENT_DATE),\n is_current_week = (EXTRACT(week FROM full_date) = EXTRACT(week FROM CURRENT_DATE)\n AND EXTRACT(year FROM full_date) = EXTRACT(year FROM CURRENT_DATE)),\n is_current_month = (EXTRACT(month FROM full_date) = EXTRACT(month FROM CURRENT_DATE)\n AND EXTRACT(year FROM full_date) = EXTRACT(year FROM CURRENT_DATE)),\n is_current_quarter = (EXTRACT(quarter FROM full_date) = EXTRACT(quarter FROM CURRENT_DATE)\n AND EXTRACT(year FROM full_date) = EXTRACT(year FROM CURRENT_DATE)),\n is_current_year = (EXTRACT(year FROM full_date) = EXTRACT(year FROM CURRENT_DATE));\nEND;\n$;\n```\n\n### Fact Table Pattern\n\n```sql\n-- ============================================================================\n-- Gold Fact: Sales\n-- ============================================================================\n\nCREATE TABLE gold.fact_sales (\n -- Surrogate key\n sales_key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n \n -- Dimension keys (foreign keys to dimensions)\n customer_key bigint NOT NULL REFERENCES gold.dim_customer(customer_key),\n product_key bigint NOT NULL REFERENCES gold.dim_product(product_key),\n date_key integer NOT NULL REFERENCES gold.dim_date(date_key),\n \n -- Degenerate dimensions (transaction details)\n order_id uuid NOT NULL,\n order_line_number smallint NOT NULL,\n \n -- Measures\n quantity integer NOT NULL,\n unit_price numeric(10,2) NOT NULL,\n discount_amount numeric(10,2) NOT NULL DEFAULT 0,\n tax_amount numeric(10,2) NOT NULL DEFAULT 0,\n line_total numeric(10,2) GENERATED ALWAYS AS (\n quantity * unit_price - discount_amount + tax_amount\n ) STORED,\n \n -- Lineage\n _silver_order_sk bigint,\n _batch_id uuid NOT NULL,\n _loaded_at timestamptz NOT NULL DEFAULT now(),\n \n -- Constraints\n CONSTRAINT fact_sales_quantity_positive CHECK (quantity > 0),\n CONSTRAINT fact_sales_price_positive CHECK (unit_price >= 0)\n);\n\n-- Partition by date for time-based queries\nCREATE TABLE gold.fact_sales_partitioned (\n sales_key bigint GENERATED ALWAYS AS IDENTITY,\n customer_key bigint NOT NULL,\n product_key bigint NOT NULL,\n date_key integer NOT NULL,\n order_id uuid NOT NULL,\n order_line_number smallint NOT NULL,\n quantity integer NOT NULL,\n unit_price numeric(10,2) NOT NULL,\n discount_amount numeric(10,2) NOT NULL DEFAULT 0,\n tax_amount numeric(10,2) NOT NULL DEFAULT 0,\n line_total numeric(10,2),\n _batch_id uuid NOT NULL,\n _loaded_at timestamptz NOT NULL DEFAULT now(),\n PRIMARY KEY (sales_key, date_key)\n) PARTITION BY RANGE (date_key);\n\n-- Create yearly partitions\nCREATE TABLE gold.fact_sales_2024 PARTITION OF gold.fact_sales_partitioned\n FOR VALUES FROM (20240101) TO (20250101);\n```\n\n### Aggregate Table Pattern\n\n```sql\n-- ============================================================================\n-- Gold Aggregate: Daily Sales\n-- ============================================================================\n\nCREATE TABLE gold.agg_daily_sales (\n date_key integer NOT NULL REFERENCES gold.dim_date(date_key),\n customer_segment text,\n product_category text,\n \n -- Aggregate measures\n order_count integer NOT NULL,\n item_count integer NOT NULL,\n gross_revenue numeric(12,2) NOT NULL,\n discount_total numeric(12,2) NOT NULL,\n tax_total numeric(12,2) NOT NULL,\n net_revenue numeric(12,2) NOT NULL,\n avg_order_value numeric(10,2) NOT NULL,\n \n -- Lineage\n _batch_id uuid NOT NULL,\n _loaded_at timestamptz NOT NULL DEFAULT now(),\n \n PRIMARY KEY (date_key, customer_segment, product_category)\n);\n\n-- Refresh aggregate\nCREATE OR REPLACE PROCEDURE dwh_etl.refresh_agg_daily_sales(\n in_date_key integer DEFAULT NULL\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_batch_id uuid := gen_random_uuid();\nBEGIN\n -- Delete existing data for the date(s)\n IF in_date_key IS NOT NULL THEN\n DELETE FROM gold.agg_daily_sales WHERE date_key = in_date_key;\n ELSE\n TRUNCATE gold.agg_daily_sales;\n END IF;\n \n -- Insert aggregated data\n INSERT INTO gold.agg_daily_sales (\n date_key, customer_segment, product_category,\n order_count, item_count, gross_revenue, discount_total, tax_total,\n net_revenue, avg_order_value, _batch_id\n )\n SELECT \n f.date_key,\n COALESCE(c.customer_segment, 'Unknown'),\n COALESCE(p.product_category, 'Unknown'),\n COUNT(DISTINCT f.order_id),\n SUM(f.quantity),\n SUM(f.quantity * f.unit_price),\n SUM(f.discount_amount),\n SUM(f.tax_amount),\n SUM(f.line_total),\n AVG(f.line_total),\n l_batch_id\n FROM gold.fact_sales f\n JOIN gold.dim_customer c ON c.customer_key = f.customer_key\n JOIN gold.dim_product p ON p.product_key = f.product_key\n WHERE (in_date_key IS NULL OR f.date_key = in_date_key)\n GROUP BY f.date_key, c.customer_segment, p.product_category;\n \n -- Log to lineage\n PERFORM dwh_lineage.log_aggregation(\n in_source_table := 'gold.fact_sales',\n in_target_table := 'gold.agg_daily_sales',\n in_batch_id := l_batch_id\n );\nEND;\n$;\n```\n\n### Materialized Views for Performance\n\n```sql\n-- Materialized view for frequently accessed aggregates\nCREATE MATERIALIZED VIEW gold.mv_customer_lifetime_value AS\nSELECT \n c.customer_key,\n c.customer_id,\n c.email,\n c.name,\n c.customer_segment,\n COUNT(DISTINCT f.order_id) AS total_orders,\n SUM(f.line_total) AS lifetime_value,\n AVG(f.line_total) AS avg_order_value,\n MIN(d.full_date) AS first_order_date,\n MAX(d.full_date) AS last_order_date,\n MAX(d.full_date) - MIN(d.full_date) AS customer_tenure_days\nFROM gold.dim_customer c\nLEFT JOIN gold.fact_sales f ON f.customer_key = c.customer_key\nLEFT JOIN gold.dim_date d ON d.date_key = f.date_key\nWHERE c.is_current = true\nGROUP BY c.customer_key, c.customer_id, c.email, c.name, c.customer_segment;\n\n-- Index for the materialized view\nCREATE UNIQUE INDEX mv_customer_ltv_pk ON gold.mv_customer_lifetime_value(customer_key);\nCREATE INDEX mv_customer_ltv_value ON gold.mv_customer_lifetime_value(lifetime_value DESC);\n\n-- Refresh procedure\nCREATE OR REPLACE PROCEDURE dwh_etl.refresh_customer_ltv()\nLANGUAGE plpgsql\nAS $\nBEGIN\n REFRESH MATERIALIZED VIEW CONCURRENTLY gold.mv_customer_lifetime_value;\n \n PERFORM dwh_lineage.log_refresh(\n in_view_name := 'gold.mv_customer_lifetime_value'\n );\nEND;\n$;\n```\n\n---\n\n## Data Lineage Tracking\n\nData lineage tracks the **origin, transformations, and destinations** of data throughout the pipeline.\n\n### Lineage Schema Design\n\n```mermaid\nerDiagram\n PIPELINE_RUNS ||--o{ TABLE_LINEAGE : \"tracks\"\n TABLE_LINEAGE ||--o{ COLUMN_LINEAGE : \"contains\"\n PIPELINE_RUNS ||--o{ QUALITY_METRICS : \"measures\"\n \n PIPELINE_RUNS {\n bigint run_id PK\n uuid batch_id UK\n text pipeline_name\n timestamptz started_at\n timestamptz completed_at\n text status\n jsonb parameters\n jsonb metrics\n }\n \n TABLE_LINEAGE {\n bigint lineage_id PK\n bigint run_id FK\n text source_table\n text target_table\n text operation\n bigint rows_read\n bigint rows_written\n timestamptz executed_at\n }\n \n COLUMN_LINEAGE {\n bigint id PK\n bigint lineage_id FK\n text source_column\n text target_column\n text transformation\n }\n \n QUALITY_METRICS {\n bigint id PK\n bigint run_id FK\n text table_name\n text metric_name\n numeric metric_value\n jsonb details\n }\n```\n\n### Lineage Tables Implementation\n\n```sql\n-- ============================================================================\n-- Data Lineage Schema\n-- ============================================================================\n\n-- Pipeline execution tracking\nCREATE TABLE dwh_lineage.pipeline_runs (\n run_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n batch_id uuid NOT NULL UNIQUE DEFAULT gen_random_uuid(),\n pipeline_name text NOT NULL,\n started_at timestamptz NOT NULL DEFAULT now(),\n completed_at timestamptz,\n status text NOT NULL DEFAULT 'running' \n CHECK (status IN ('running', 'completed', 'failed', 'cancelled')),\n parameters jsonb DEFAULT '{}',\n error_message text,\n metrics jsonb DEFAULT '{}'\n);\n\nCREATE INDEX pipeline_runs_name_idx ON dwh_lineage.pipeline_runs(pipeline_name);\nCREATE INDEX pipeline_runs_status_idx ON dwh_lineage.pipeline_runs(status);\nCREATE INDEX pipeline_runs_started_idx ON dwh_lineage.pipeline_runs(started_at);\n\n-- Table-level lineage\nCREATE TABLE dwh_lineage.table_lineage (\n lineage_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n run_id bigint REFERENCES dwh_lineage.pipeline_runs(run_id),\n batch_id uuid NOT NULL,\n \n -- Source and target\n source_table text NOT NULL,\n target_table text NOT NULL,\n operation text NOT NULL CHECK (operation IN (\n 'ingest', 'transform', 'aggregate', 'refresh', 'delete'\n )),\n \n -- Metrics\n rows_read bigint,\n rows_inserted bigint,\n rows_updated bigint,\n rows_deleted bigint,\n rows_rejected bigint,\n \n -- Timing\n started_at timestamptz NOT NULL DEFAULT now(),\n completed_at timestamptz,\n duration_ms integer GENERATED ALWAYS AS (\n EXTRACT(MILLISECONDS FROM (completed_at - started_at))::integer\n ) STORED,\n \n -- Additional context\n sql_hash text, -- Hash of executed SQL\n notes text\n);\n\nCREATE INDEX table_lineage_source_idx ON dwh_lineage.table_lineage(source_table);\nCREATE INDEX table_lineage_target_idx ON dwh_lineage.table_lineage(target_table);\nCREATE INDEX table_lineage_batch_idx ON dwh_lineage.table_lineage(batch_id);\n\n-- Column-level lineage (optional, for detailed tracking)\nCREATE TABLE dwh_lineage.column_lineage (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n lineage_id bigint REFERENCES dwh_lineage.table_lineage(lineage_id),\n source_column text NOT NULL,\n target_column text NOT NULL,\n transformation text, -- e.g., 'lower(trim($1))'\n is_key boolean DEFAULT false,\n notes text\n);\n\n-- Data quality metrics\nCREATE TABLE dwh_lineage.quality_metrics (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n run_id bigint REFERENCES dwh_lineage.pipeline_runs(run_id),\n batch_id uuid NOT NULL,\n table_name text NOT NULL,\n metric_name text NOT NULL,\n metric_value numeric,\n threshold numeric,\n passed boolean,\n details jsonb,\n measured_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Dependency graph (for scheduling and impact analysis)\nCREATE TABLE dwh_lineage.table_dependencies (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n upstream_table text NOT NULL,\n downstream_table text NOT NULL,\n dependency_type text NOT NULL DEFAULT 'data' \n CHECK (dependency_type IN ('data', 'reference', 'optional')),\n created_at timestamptz NOT NULL DEFAULT now(),\n UNIQUE (upstream_table, downstream_table)\n);\n```\n\n### Lineage Helper Functions\n\n```sql\n-- ============================================================================\n-- Lineage Logging Functions\n-- ============================================================================\n\n-- Start a pipeline run\nCREATE OR REPLACE FUNCTION dwh_lineage.start_pipeline(\n in_pipeline_name text,\n in_parameters jsonb DEFAULT '{}'\n)\nRETURNS uuid\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_batch_id uuid := gen_random_uuid();\nBEGIN\n INSERT INTO dwh_lineage.pipeline_runs (batch_id, pipeline_name, parameters)\n VALUES (l_batch_id, in_pipeline_name, in_parameters);\n \n RETURN l_batch_id;\nEND;\n$;\n\n-- Complete a pipeline run\nCREATE OR REPLACE PROCEDURE dwh_lineage.complete_pipeline(\n in_batch_id uuid,\n in_status text DEFAULT 'completed',\n in_error_message text DEFAULT NULL,\n in_metrics jsonb DEFAULT '{}'\n)\nLANGUAGE plpgsql\nAS $\nBEGIN\n UPDATE dwh_lineage.pipeline_runs\n SET completed_at = now(),\n status = in_status,\n error_message = in_error_message,\n metrics = in_metrics\n WHERE batch_id = in_batch_id;\nEND;\n$;\n\n-- Log ingestion (bronze layer)\nCREATE OR REPLACE FUNCTION dwh_lineage.log_ingestion(\n in_table_name text,\n in_batch_id uuid,\n in_source text,\n in_row_count integer\n)\nRETURNS bigint\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_lineage_id bigint;\nBEGIN\n INSERT INTO dwh_lineage.table_lineage (\n batch_id, source_table, target_table, operation,\n rows_read, rows_inserted, completed_at\n ) VALUES (\n in_batch_id, in_source, in_table_name, 'ingest',\n in_row_count, in_row_count, now()\n )\n RETURNING lineage_id INTO l_lineage_id;\n \n RETURN l_lineage_id;\nEND;\n$;\n\n-- Log transformation (silver/gold layers)\nCREATE OR REPLACE FUNCTION dwh_lineage.log_transformation(\n in_source_table text,\n in_target_table text,\n in_batch_id uuid,\n in_rows_inserted integer DEFAULT 0,\n in_rows_updated integer DEFAULT 0,\n in_rows_unchanged integer DEFAULT 0,\n in_rows_rejected integer DEFAULT 0\n)\nRETURNS bigint\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_lineage_id bigint;\nBEGIN\n INSERT INTO dwh_lineage.table_lineage (\n batch_id, source_table, target_table, operation,\n rows_read, rows_inserted, rows_updated, rows_rejected,\n completed_at\n ) VALUES (\n in_batch_id, in_source_table, in_target_table, 'transform',\n in_rows_inserted + in_rows_updated + in_rows_unchanged + in_rows_rejected,\n in_rows_inserted, in_rows_updated, in_rows_rejected,\n now()\n )\n RETURNING lineage_id INTO l_lineage_id;\n \n -- Update/create dependency\n INSERT INTO dwh_lineage.table_dependencies (upstream_table, downstream_table)\n VALUES (in_source_table, in_target_table)\n ON CONFLICT (upstream_table, downstream_table) DO NOTHING;\n \n RETURN l_lineage_id;\nEND;\n$;\n\n-- Log aggregation\nCREATE OR REPLACE FUNCTION dwh_lineage.log_aggregation(\n in_source_table text,\n in_target_table text,\n in_batch_id uuid\n)\nRETURNS bigint\nLANGUAGE plpgsql\nAS $\nBEGIN\n RETURN dwh_lineage.log_transformation(\n in_source_table := in_source_table,\n in_target_table := in_target_table,\n in_batch_id := in_batch_id\n );\nEND;\n$;\n\n-- Log materialized view refresh\nCREATE OR REPLACE FUNCTION dwh_lineage.log_refresh(\n in_view_name text\n)\nRETURNS bigint\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_lineage_id bigint;\nBEGIN\n INSERT INTO dwh_lineage.table_lineage (\n batch_id, source_table, target_table, operation, completed_at\n ) VALUES (\n gen_random_uuid(), 'multiple', in_view_name, 'refresh', now()\n )\n RETURNING lineage_id INTO l_lineage_id;\n \n RETURN l_lineage_id;\nEND;\n$;\n```\n\n### Lineage Query Functions\n\n```sql\n-- ============================================================================\n-- Lineage Query Functions\n-- ============================================================================\n\n-- Get upstream dependencies (what feeds this table?)\nCREATE OR REPLACE FUNCTION dwh_lineage.get_upstream(in_table_name text)\nRETURNS TABLE (\n level int,\n table_name text,\n dependency_type text\n)\nLANGUAGE sql\nSTABLE\nAS $\n WITH RECURSIVE upstream AS (\n SELECT 1 AS level, upstream_table AS table_name, dependency_type\n FROM dwh_lineage.table_dependencies\n WHERE downstream_table = in_table_name\n \n UNION ALL\n \n SELECT u.level + 1, d.upstream_table, d.dependency_type\n FROM upstream u\n JOIN dwh_lineage.table_dependencies d ON d.downstream_table = u.table_name\n WHERE u.level \u003c 10 -- Prevent infinite loops\n )\n SELECT DISTINCT level, table_name, dependency_type\n FROM upstream\n ORDER BY level, table_name;\n$;\n\n-- Get downstream dependencies (what does this table feed?)\nCREATE OR REPLACE FUNCTION dwh_lineage.get_downstream(in_table_name text)\nRETURNS TABLE (\n level int,\n table_name text,\n dependency_type text\n)\nLANGUAGE sql\nSTABLE\nAS $\n WITH RECURSIVE downstream AS (\n SELECT 1 AS level, downstream_table AS table_name, dependency_type\n FROM dwh_lineage.table_dependencies\n WHERE upstream_table = in_table_name\n \n UNION ALL\n \n SELECT d.level + 1, dep.downstream_table, dep.dependency_type\n FROM downstream d\n JOIN dwh_lineage.table_dependencies dep ON dep.upstream_table = d.table_name\n WHERE d.level \u003c 10\n )\n SELECT DISTINCT level, table_name, dependency_type\n FROM downstream\n ORDER BY level, table_name;\n$;\n\n-- Get complete lineage for a record\nCREATE OR REPLACE FUNCTION dwh_lineage.trace_record(\n in_table_name text,\n in_batch_id uuid\n)\nRETURNS TABLE (\n step int,\n operation text,\n source_table text,\n target_table text,\n executed_at timestamptz,\n rows_affected bigint\n)\nLANGUAGE sql\nSTABLE\nAS $\n WITH RECURSIVE lineage_trace AS (\n -- Start from the target\n SELECT \n 1 AS step,\n tl.operation,\n tl.source_table,\n tl.target_table,\n tl.completed_at,\n COALESCE(tl.rows_inserted, 0) + COALESCE(tl.rows_updated, 0) AS rows_affected,\n tl.source_table AS next_target\n FROM dwh_lineage.table_lineage tl\n WHERE tl.target_table = in_table_name\n AND tl.batch_id = in_batch_id\n \n UNION ALL\n \n -- Trace back through sources\n SELECT \n lt.step + 1,\n tl.operation,\n tl.source_table,\n tl.target_table,\n tl.completed_at,\n COALESCE(tl.rows_inserted, 0) + COALESCE(tl.rows_updated, 0),\n tl.source_table\n FROM lineage_trace lt\n JOIN dwh_lineage.table_lineage tl ON tl.target_table = lt.next_target\n WHERE lt.step \u003c 10\n AND tl.completed_at \u003c lt.completed_at -- Earlier in time\n )\n SELECT step, operation, source_table, target_table, completed_at, rows_affected\n FROM lineage_trace\n ORDER BY step;\n$;\n\n-- Pipeline execution history\nCREATE OR REPLACE FUNCTION dwh_lineage.get_pipeline_history(\n in_pipeline_name text,\n in_limit integer DEFAULT 10\n)\nRETURNS TABLE (\n batch_id uuid,\n started_at timestamptz,\n completed_at timestamptz,\n duration_seconds numeric,\n status text,\n tables_processed bigint,\n total_rows bigint\n)\nLANGUAGE sql\nSTABLE\nAS $\n SELECT \n pr.batch_id,\n pr.started_at,\n pr.completed_at,\n EXTRACT(EPOCH FROM (pr.completed_at - pr.started_at)),\n pr.status,\n COUNT(DISTINCT tl.target_table),\n SUM(COALESCE(tl.rows_inserted, 0) + COALESCE(tl.rows_updated, 0))\n FROM dwh_lineage.pipeline_runs pr\n LEFT JOIN dwh_lineage.table_lineage tl ON tl.batch_id = pr.batch_id\n WHERE pr.pipeline_name = in_pipeline_name\n GROUP BY pr.batch_id, pr.started_at, pr.completed_at, pr.status\n ORDER BY pr.started_at DESC\n LIMIT in_limit;\n$;\n```\n\n### Lineage Visualization View\n\n```sql\n-- View for building lineage diagrams\nCREATE OR REPLACE VIEW dwh_lineage.v_dependency_graph AS\nSELECT \n upstream_table AS source,\n downstream_table AS target,\n dependency_type,\n CASE \n WHEN upstream_table LIKE 'bronze.%' THEN 'bronze'\n WHEN upstream_table LIKE 'silver.%' THEN 'silver'\n WHEN upstream_table LIKE 'gold.%' THEN 'gold'\n ELSE 'external'\n END AS source_layer,\n CASE \n WHEN downstream_table LIKE 'bronze.%' THEN 'bronze'\n WHEN downstream_table LIKE 'silver.%' THEN 'silver'\n WHEN downstream_table LIKE 'gold.%' THEN 'gold'\n ELSE 'external'\n END AS target_layer\nFROM dwh_lineage.table_dependencies\nORDER BY source_layer, target_layer, source, target;\n```\n\n---\n\n## ETL Orchestration\n\n### Pipeline Execution Pattern\n\n```mermaid\nsequenceDiagram\n participant Scheduler\n participant Pipeline as dwh_etl.run_daily_pipeline()\n participant Bronze\n participant Silver\n participant Gold\n participant Lineage as dwh_lineage\n \n Scheduler->>Pipeline: Trigger daily run\n Pipeline->>Lineage: start_pipeline()\n activate Pipeline\n \n Pipeline->>Bronze: Load raw data\n Bronze->>Lineage: log_ingestion()\n \n Pipeline->>Silver: Transform data\n Silver->>Lineage: log_transformation()\n \n Pipeline->>Gold: Build aggregates\n Gold->>Lineage: log_aggregation()\n \n Pipeline->>Lineage: complete_pipeline()\n deactivate Pipeline\n \n Pipeline-->>Scheduler: Return status\n```\n\n### Master Pipeline Procedure\n\n```sql\n-- ============================================================================\n-- Daily ETL Pipeline\n-- ============================================================================\n\nCREATE OR REPLACE PROCEDURE dwh_etl.run_daily_pipeline(\n in_date date DEFAULT CURRENT_DATE - 1\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_batch_id uuid;\n l_step text;\n l_start_time timestamptz;\n l_metrics jsonb := '{}';\nBEGIN\n l_start_time := clock_timestamp();\n \n -- Start pipeline tracking\n l_batch_id := dwh_lineage.start_pipeline(\n 'daily_etl',\n jsonb_build_object('date', in_date)\n );\n \n RAISE NOTICE 'Starting daily pipeline for % (batch: %)', in_date, l_batch_id;\n \n BEGIN\n -- ========================================\n -- BRONZE: Ingest raw data\n -- ========================================\n l_step := 'bronze_customers';\n RAISE NOTICE 'Step: %', l_step;\n CALL dwh_etl.ingest_daily_customers(in_date, l_batch_id);\n \n l_step := 'bronze_orders';\n RAISE NOTICE 'Step: %', l_step;\n CALL dwh_etl.ingest_daily_orders(in_date, l_batch_id);\n \n -- ========================================\n -- SILVER: Cleanse and validate\n -- ========================================\n l_step := 'silver_customers';\n RAISE NOTICE 'Step: %', l_step;\n CALL dwh_etl.load_silver_customers(l_batch_id);\n \n l_step := 'silver_orders';\n RAISE NOTICE 'Step: %', l_step;\n CALL dwh_etl.load_silver_orders(l_batch_id);\n \n -- ========================================\n -- GOLD: Build dimensions and facts\n -- ========================================\n l_step := 'gold_dim_customer';\n RAISE NOTICE 'Step: %', l_step;\n CALL dwh_etl.load_dim_customer(l_batch_id);\n \n l_step := 'gold_fact_sales';\n RAISE NOTICE 'Step: %', l_step;\n CALL dwh_etl.load_fact_sales(in_date, l_batch_id);\n \n -- ========================================\n -- AGGREGATES: Refresh summaries\n -- ========================================\n l_step := 'agg_daily_sales';\n RAISE NOTICE 'Step: %', l_step;\n CALL dwh_etl.refresh_agg_daily_sales(\n to_char(in_date, 'YYYYMMDD')::integer\n );\n \n l_step := 'mv_customer_ltv';\n RAISE NOTICE 'Step: %', l_step;\n CALL dwh_etl.refresh_customer_ltv();\n \n -- Calculate metrics\n l_metrics := jsonb_build_object(\n 'duration_seconds', EXTRACT(EPOCH FROM (clock_timestamp() - l_start_time)),\n 'date_processed', in_date\n );\n \n -- Complete successfully\n CALL dwh_lineage.complete_pipeline(l_batch_id, 'completed', NULL, l_metrics);\n \n RAISE NOTICE 'Pipeline completed successfully in % seconds', \n EXTRACT(EPOCH FROM (clock_timestamp() - l_start_time));\n \n EXCEPTION\n WHEN OTHERS THEN\n -- Log failure\n CALL dwh_lineage.complete_pipeline(\n l_batch_id, \n 'failed', \n format('Step %s failed: %s', l_step, SQLERRM),\n l_metrics\n );\n \n RAISE EXCEPTION 'Pipeline failed at step %: %', l_step, SQLERRM;\n END;\nEND;\n$;\n```\n\n### Dependency-Based Execution\n\n```sql\n-- ============================================================================\n-- Execute Tables in Dependency Order\n-- ============================================================================\n\nCREATE OR REPLACE PROCEDURE dwh_etl.run_pipeline_by_dependencies(\n in_target_table text,\n in_batch_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_batch_id uuid;\n l_table RECORD;\n l_proc_name text;\nBEGIN\n l_batch_id := COALESCE(in_batch_id, gen_random_uuid());\n \n -- Get tables in dependency order\n FOR l_table IN\n WITH RECURSIVE deps AS (\n SELECT 0 AS level, in_target_table AS table_name\n \n UNION ALL\n \n SELECT d.level + 1, td.upstream_table\n FROM deps d\n JOIN dwh_lineage.table_dependencies td ON td.downstream_table = d.table_name\n WHERE d.level \u003c 10\n )\n SELECT DISTINCT table_name, MAX(level) AS execution_order\n FROM deps\n GROUP BY table_name\n ORDER BY MAX(level) DESC, table_name\n LOOP\n -- Derive procedure name from table name\n l_proc_name := 'dwh_etl.load_' || replace(l_table.table_name, '.', '_');\n \n RAISE NOTICE 'Executing: % (order: %)', l_proc_name, l_table.execution_order;\n \n -- Execute if procedure exists\n IF EXISTS (\n SELECT 1 FROM pg_proc p\n JOIN pg_namespace n ON p.pronamespace = n.oid\n WHERE n.nspname || '.' || p.proname = l_proc_name\n ) THEN\n EXECUTE format('CALL %s($1)', l_proc_name) USING l_batch_id;\n ELSE\n RAISE NOTICE 'Procedure not found: %', l_proc_name;\n END IF;\n END LOOP;\nEND;\n$;\n```\n\n---\n\n## Incremental Processing\n\n### Watermark-Based Incremental Load\n\n```sql\n-- ============================================================================\n-- Watermark Tracking\n-- ============================================================================\n\nCREATE TABLE dwh_etl.watermarks (\n table_name text PRIMARY KEY,\n watermark_column text NOT NULL,\n watermark_value timestamptz NOT NULL,\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Get watermark\nCREATE OR REPLACE FUNCTION dwh_etl.get_watermark(in_table_name text)\nRETURNS timestamptz\nLANGUAGE sql\nSTABLE\nAS $\n SELECT COALESCE(watermark_value, '1900-01-01'::timestamptz)\n FROM dwh_etl.watermarks\n WHERE table_name = in_table_name;\n$;\n\n-- Update watermark\nCREATE OR REPLACE PROCEDURE dwh_etl.set_watermark(\n in_table_name text,\n in_column text,\n in_value timestamptz\n)\nLANGUAGE plpgsql\nAS $\nBEGIN\n INSERT INTO dwh_etl.watermarks (table_name, watermark_column, watermark_value)\n VALUES (in_table_name, in_column, in_value)\n ON CONFLICT (table_name) DO UPDATE SET\n watermark_value = EXCLUDED.watermark_value,\n updated_at = now();\nEND;\n$;\n\n-- Incremental load example\nCREATE OR REPLACE PROCEDURE dwh_etl.load_silver_orders_incremental(\n in_batch_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_batch_id uuid;\n l_watermark timestamptz;\n l_new_watermark timestamptz;\n l_row_count integer;\nBEGIN\n l_batch_id := COALESCE(in_batch_id, gen_random_uuid());\n l_watermark := dwh_etl.get_watermark('silver.orders');\n \n -- Find max timestamp in new data\n SELECT MAX(_ingested_at) INTO l_new_watermark\n FROM bronze.raw_orders\n WHERE _ingested_at > l_watermark;\n \n IF l_new_watermark IS NULL THEN\n RAISE NOTICE 'No new data to process';\n RETURN;\n END IF;\n \n -- Process only new records\n INSERT INTO silver.orders (...)\n SELECT ...\n FROM bronze.raw_orders\n WHERE _ingested_at > l_watermark\n AND _ingested_at \u003c= l_new_watermark;\n \n GET DIAGNOSTICS l_row_count = ROW_COUNT;\n \n -- Update watermark\n CALL dwh_etl.set_watermark('silver.orders', '_ingested_at', l_new_watermark);\n \n -- Log lineage\n PERFORM dwh_lineage.log_transformation(\n 'bronze.raw_orders', 'silver.orders', l_batch_id,\n in_rows_inserted := l_row_count\n );\n \n RAISE NOTICE 'Processed % rows (watermark: % -> %)', \n l_row_count, l_watermark, l_new_watermark;\nEND;\n$;\n```\n\n### Merge Pattern for Incremental Updates\n\n```sql\n-- ============================================================================\n-- Incremental Merge (Upsert) Pattern\n-- ============================================================================\n\nCREATE OR REPLACE PROCEDURE dwh_etl.merge_dim_customer(\n in_batch_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_batch_id uuid;\n l_inserted integer := 0;\n l_updated integer := 0;\nBEGIN\n l_batch_id := COALESCE(in_batch_id, gen_random_uuid());\n \n -- Create staging with changes\n CREATE TEMP TABLE stg_dim_customer AS\n SELECT \n s.customer_sk AS silver_sk,\n s.customer_id,\n s.email,\n s.name,\n s.status,\n s.valid_from AS source_valid_from,\n md5(s.email || s.name || s.status) AS row_hash\n FROM silver.customers s\n WHERE s.is_current = true\n AND s._loaded_at > dwh_etl.get_watermark('gold.dim_customer');\n \n -- Update existing (Type 1 - overwrite)\n UPDATE gold.dim_customer g\n SET email = s.email,\n name = s.name,\n status = s.status,\n _silver_sk = s.silver_sk,\n _batch_id = l_batch_id,\n _loaded_at = now()\n FROM stg_dim_customer s\n WHERE g.customer_id = s.customer_id\n AND g.is_current = true\n AND md5(g.email || g.name || g.status) != s.row_hash;\n \n GET DIAGNOSTICS l_updated = ROW_COUNT;\n \n -- Insert new\n INSERT INTO gold.dim_customer (\n customer_id, email, name, status,\n valid_from, is_current, _silver_sk, _batch_id\n )\n SELECT \n s.customer_id, s.email, s.name, s.status,\n s.source_valid_from, true, s.silver_sk, l_batch_id\n FROM stg_dim_customer s\n WHERE NOT EXISTS (\n SELECT 1 FROM gold.dim_customer g \n WHERE g.customer_id = s.customer_id\n );\n \n GET DIAGNOSTICS l_inserted = ROW_COUNT;\n \n -- Update watermark\n CALL dwh_etl.set_watermark(\n 'gold.dim_customer', \n '_loaded_at',\n (SELECT MAX(_loaded_at) FROM silver.customers)\n );\n \n -- Cleanup\n DROP TABLE stg_dim_customer;\n \n -- Log\n PERFORM dwh_lineage.log_transformation(\n 'silver.customers', 'gold.dim_customer', l_batch_id,\n l_inserted, l_updated\n );\n \n RAISE NOTICE 'Dim customer: % inserted, % updated', l_inserted, l_updated;\nEND;\n$;\n```\n\n---\n\n## Data Quality Framework\n\n### Quality Check Functions\n\n```sql\n-- ============================================================================\n-- Data Quality Checks\n-- ============================================================================\n\n-- Generic null check\nCREATE OR REPLACE FUNCTION dwh_etl.check_not_null(\n in_table text,\n in_column text,\n in_batch_id uuid DEFAULT NULL\n)\nRETURNS TABLE (passed boolean, null_count bigint, total_count bigint)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_null_count bigint;\n l_total_count bigint;\nBEGIN\n EXECUTE format(\n 'SELECT COUNT(*) FILTER (WHERE %I IS NULL), COUNT(*) FROM %s',\n in_column, in_table\n ) INTO l_null_count, l_total_count;\n \n IF in_batch_id IS NOT NULL THEN\n INSERT INTO dwh_lineage.quality_metrics (\n batch_id, table_name, metric_name, metric_value, threshold, passed, details\n ) VALUES (\n in_batch_id, in_table, 'null_check_' || in_column, \n l_null_count, 0, l_null_count = 0,\n jsonb_build_object('column', in_column, 'null_count', l_null_count)\n );\n END IF;\n \n RETURN QUERY SELECT l_null_count = 0, l_null_count, l_total_count;\nEND;\n$;\n\n-- Uniqueness check\nCREATE OR REPLACE FUNCTION dwh_etl.check_unique(\n in_table text,\n in_columns text[],\n in_batch_id uuid DEFAULT NULL\n)\nRETURNS TABLE (passed boolean, duplicate_count bigint)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_column_list text;\n l_dup_count bigint;\nBEGIN\n l_column_list := array_to_string(in_columns, ', ');\n \n EXECUTE format(\n 'SELECT COUNT(*) FROM (\n SELECT %s FROM %s GROUP BY %s HAVING COUNT(*) > 1\n ) dups',\n l_column_list, in_table, l_column_list\n ) INTO l_dup_count;\n \n IF in_batch_id IS NOT NULL THEN\n INSERT INTO dwh_lineage.quality_metrics (\n batch_id, table_name, metric_name, metric_value, threshold, passed, details\n ) VALUES (\n in_batch_id, in_table, 'unique_check', \n l_dup_count, 0, l_dup_count = 0,\n jsonb_build_object('columns', in_columns, 'duplicate_groups', l_dup_count)\n );\n END IF;\n \n RETURN QUERY SELECT l_dup_count = 0, l_dup_count;\nEND;\n$;\n\n-- Referential integrity check\nCREATE OR REPLACE FUNCTION dwh_etl.check_referential_integrity(\n in_child_table text,\n in_child_column text,\n in_parent_table text,\n in_parent_column text,\n in_batch_id uuid DEFAULT NULL\n)\nRETURNS TABLE (passed boolean, orphan_count bigint)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_orphan_count bigint;\nBEGIN\n EXECUTE format(\n 'SELECT COUNT(*) FROM %s c \n WHERE NOT EXISTS (SELECT 1 FROM %s p WHERE p.%I = c.%I)\n AND c.%I IS NOT NULL',\n in_child_table, in_parent_table, \n in_parent_column, in_child_column, in_child_column\n ) INTO l_orphan_count;\n \n IF in_batch_id IS NOT NULL THEN\n INSERT INTO dwh_lineage.quality_metrics (\n batch_id, table_name, metric_name, metric_value, threshold, passed, details\n ) VALUES (\n in_batch_id, in_child_table, 'referential_integrity', \n l_orphan_count, 0, l_orphan_count = 0,\n jsonb_build_object(\n 'child_column', in_child_column,\n 'parent_table', in_parent_table,\n 'parent_column', in_parent_column\n )\n );\n END IF;\n \n RETURN QUERY SELECT l_orphan_count = 0, l_orphan_count;\nEND;\n$;\n\n-- Value range check\nCREATE OR REPLACE FUNCTION dwh_etl.check_range(\n in_table text,\n in_column text,\n in_min numeric,\n in_max numeric,\n in_batch_id uuid DEFAULT NULL\n)\nRETURNS TABLE (passed boolean, out_of_range_count bigint)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_oor_count bigint;\nBEGIN\n EXECUTE format(\n 'SELECT COUNT(*) FROM %s WHERE %I \u003c %s OR %I > %s',\n in_table, in_column, in_min, in_column, in_max\n ) INTO l_oor_count;\n \n IF in_batch_id IS NOT NULL THEN\n INSERT INTO dwh_lineage.quality_metrics (\n batch_id, table_name, metric_name, metric_value, threshold, passed, details\n ) VALUES (\n in_batch_id, in_table, 'range_check_' || in_column, \n l_oor_count, 0, l_oor_count = 0,\n jsonb_build_object('column', in_column, 'min', in_min, 'max', in_max)\n );\n END IF;\n \n RETURN QUERY SELECT l_oor_count = 0, l_oor_count;\nEND;\n$;\n```\n\n### Quality Gate Procedure\n\n```sql\n-- Run all quality checks for a table\nCREATE OR REPLACE PROCEDURE dwh_etl.run_quality_gate(\n in_table text,\n in_batch_id uuid,\n in_fail_on_error boolean DEFAULT true\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_failures integer := 0;\n l_result RECORD;\nBEGIN\n -- Define checks per table (could be driven by metadata table)\n CASE in_table\n WHEN 'silver.customers' THEN\n SELECT * INTO l_result FROM dwh_etl.check_not_null(in_table, 'customer_id', in_batch_id);\n IF NOT l_result.passed THEN l_failures := l_failures + 1; END IF;\n \n SELECT * INTO l_result FROM dwh_etl.check_not_null(in_table, 'email', in_batch_id);\n IF NOT l_result.passed THEN l_failures := l_failures + 1; END IF;\n \n SELECT * INTO l_result FROM dwh_etl.check_unique(in_table, ARRAY['customer_id'], in_batch_id);\n IF NOT l_result.passed THEN l_failures := l_failures + 1; END IF;\n \n WHEN 'gold.fact_sales' THEN\n SELECT * INTO l_result FROM dwh_etl.check_referential_integrity(\n in_table, 'customer_key', 'gold.dim_customer', 'customer_key', in_batch_id\n );\n IF NOT l_result.passed THEN l_failures := l_failures + 1; END IF;\n \n SELECT * INTO l_result FROM dwh_etl.check_range(in_table, 'quantity', 0, 10000, in_batch_id);\n IF NOT l_result.passed THEN l_failures := l_failures + 1; END IF;\n \n ELSE\n RAISE NOTICE 'No quality checks defined for %', in_table;\n END CASE;\n \n IF l_failures > 0 AND in_fail_on_error THEN\n RAISE EXCEPTION 'Quality gate failed for %: % check(s) failed', in_table, l_failures;\n ELSIF l_failures > 0 THEN\n RAISE WARNING 'Quality gate warning for %: % check(s) failed', in_table, l_failures;\n ELSE\n RAISE NOTICE 'Quality gate passed for %', in_table;\n END IF;\nEND;\n$;\n```\n\n---\n\n## Performance Optimization\n\n### Partitioning Strategies\n\n```sql\n-- ============================================================================\n-- Fact Table Partitioning\n-- ============================================================================\n\n-- Partition by date (most common for facts)\nCREATE TABLE gold.fact_sales_partitioned (\n sales_key bigint GENERATED ALWAYS AS IDENTITY,\n date_key integer NOT NULL,\n customer_key bigint NOT NULL,\n product_key bigint NOT NULL,\n quantity integer NOT NULL,\n amount numeric(10,2) NOT NULL,\n PRIMARY KEY (sales_key, date_key)\n) PARTITION BY RANGE (date_key);\n\n-- Automatic partition creation\nCREATE OR REPLACE PROCEDURE dwh_etl.create_fact_partitions(\n in_start_year integer,\n in_end_year integer\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_year integer;\n l_partition_name text;\nBEGIN\n FOR l_year IN in_start_year..in_end_year LOOP\n l_partition_name := 'gold.fact_sales_' || l_year;\n \n IF NOT EXISTS (\n SELECT 1 FROM pg_tables \n WHERE schemaname = 'gold' \n AND tablename = 'fact_sales_' || l_year\n ) THEN\n EXECUTE format(\n 'CREATE TABLE %s PARTITION OF gold.fact_sales_partitioned\n FOR VALUES FROM (%s) TO (%s)',\n l_partition_name,\n l_year * 10000 + 101, -- YYYYMMDD start\n (l_year + 1) * 10000 + 101 -- YYYYMMDD end\n );\n \n RAISE NOTICE 'Created partition: %', l_partition_name;\n END IF;\n END LOOP;\nEND;\n$;\n```\n\n### Indexing Strategies\n\n```sql\n-- ============================================================================\n-- Data Warehouse Index Patterns\n-- ============================================================================\n\n-- Dimension indexes (for lookups)\nCREATE INDEX dim_customer_email_idx ON gold.dim_customer(email) WHERE is_current;\nCREATE INDEX dim_customer_segment_idx ON gold.dim_customer(customer_segment) WHERE is_current;\n\n-- Fact table indexes (for aggregations)\nCREATE INDEX fact_sales_date_idx ON gold.fact_sales(date_key);\nCREATE INDEX fact_sales_customer_idx ON gold.fact_sales(customer_key);\nCREATE INDEX fact_sales_product_idx ON gold.fact_sales(product_key);\n\n-- Composite indexes for common query patterns\nCREATE INDEX fact_sales_date_customer_idx ON gold.fact_sales(date_key, customer_key);\n\n-- BRIN indexes for very large fact tables (ordered by date)\nCREATE INDEX fact_sales_date_brin_idx ON gold.fact_sales USING brin(date_key);\n\n-- Covering indexes for aggregate queries\nCREATE INDEX fact_sales_covering_idx ON gold.fact_sales(date_key) \n INCLUDE (quantity, amount);\n```\n\n### Statistics and Maintenance\n\n```sql\n-- Keep statistics fresh for query planner\nCREATE OR REPLACE PROCEDURE dwh_etl.maintain_statistics()\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Analyze medallion tables\n ANALYZE bronze.raw_customers;\n ANALYZE bronze.raw_orders;\n ANALYZE silver.customers;\n ANALYZE silver.orders;\n ANALYZE gold.dim_customer;\n ANALYZE gold.fact_sales;\n \n -- Vacuum large tables\n VACUUM (ANALYZE) gold.fact_sales;\n \n RAISE NOTICE 'Statistics updated';\nEND;\n$;\n```\n\n---\n\n## Complete Implementation Example\n\n### Full Pipeline Example\n\n```sql\n-- ============================================================================\n-- Complete E-Commerce Data Warehouse Implementation\n-- ============================================================================\n\n-- 1. Schema setup (run once)\n-- [Include schema creation from earlier sections]\n\n-- 2. Bronze tables\nCREATE TABLE bronze.raw_customers (\n _bronze_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n _ingested_at timestamptz NOT NULL DEFAULT now(),\n _source_system text NOT NULL,\n _batch_id uuid,\n id text,\n email text,\n name text,\n status text,\n created_date text\n);\n\nCREATE TABLE bronze.raw_orders (\n _bronze_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n _ingested_at timestamptz NOT NULL DEFAULT now(),\n _source_system text NOT NULL,\n _batch_id uuid,\n id text,\n customer_id text,\n order_date text,\n status text,\n total text\n);\n\n-- 3. Silver tables\nCREATE TABLE silver.customers (\n customer_sk bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n customer_id uuid NOT NULL,\n email text NOT NULL,\n name text NOT NULL,\n status text NOT NULL,\n valid_from timestamptz NOT NULL DEFAULT now(),\n valid_to timestamptz,\n is_current boolean NOT NULL DEFAULT true,\n _source_bronze_id bigint,\n _batch_id uuid NOT NULL,\n _loaded_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE UNIQUE INDEX silver_customers_bk ON silver.customers(customer_id) WHERE is_current;\n\n-- 4. Gold tables\nCREATE TABLE gold.dim_customer (\n customer_key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n customer_id uuid NOT NULL,\n email text NOT NULL,\n name text NOT NULL,\n status text NOT NULL,\n customer_segment text,\n valid_from timestamptz NOT NULL,\n valid_to timestamptz,\n is_current boolean NOT NULL DEFAULT true,\n _silver_sk bigint,\n _batch_id uuid NOT NULL,\n _loaded_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE TABLE gold.dim_date (\n date_key integer PRIMARY KEY,\n full_date date NOT NULL UNIQUE,\n year smallint NOT NULL,\n quarter smallint NOT NULL,\n month smallint NOT NULL,\n day_of_month smallint NOT NULL,\n month_name text NOT NULL,\n is_weekend boolean NOT NULL\n);\n\nCREATE TABLE gold.fact_sales (\n sales_key bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n date_key integer NOT NULL REFERENCES gold.dim_date(date_key),\n customer_key bigint NOT NULL REFERENCES gold.dim_customer(customer_key),\n order_id uuid NOT NULL,\n order_total numeric(10,2) NOT NULL,\n _batch_id uuid NOT NULL,\n _loaded_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- 5. Daily pipeline\nCREATE OR REPLACE PROCEDURE dwh_etl.run_ecommerce_pipeline(\n in_date date DEFAULT CURRENT_DATE - 1\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_batch_id uuid;\nBEGIN\n l_batch_id := dwh_lineage.start_pipeline('ecommerce_daily', jsonb_build_object('date', in_date));\n \n BEGIN\n -- Bronze: Ingest\n CALL dwh_etl.ingest_customers(l_batch_id);\n CALL dwh_etl.ingest_orders(in_date, l_batch_id);\n \n -- Silver: Cleanse\n CALL dwh_etl.load_silver_customers(l_batch_id);\n CALL dwh_etl.load_silver_orders(l_batch_id);\n \n -- Gold: Build\n CALL dwh_etl.load_dim_customer(l_batch_id);\n CALL dwh_etl.load_fact_sales(in_date, l_batch_id);\n \n -- Quality gate\n CALL dwh_etl.run_quality_gate('gold.fact_sales', l_batch_id, true);\n \n -- Complete\n CALL dwh_lineage.complete_pipeline(l_batch_id, 'completed');\n \n EXCEPTION\n WHEN OTHERS THEN\n CALL dwh_lineage.complete_pipeline(l_batch_id, 'failed', SQLERRM);\n RAISE;\n END;\nEND;\n$;\n\n-- 6. Schedule with pg_cron\nSELECT cron.schedule(\n 'daily-dwh-pipeline',\n '0 2 * * *', -- 2 AM daily\n 'CALL dwh_etl.run_ecommerce_pipeline()'\n);\n```\n\n---\n\n## Summary\n\n### Key Takeaways\n\n1. **Schema Organization**: Use separate schemas for each medallion layer plus dedicated lineage and ETL schemas\n2. **Bronze Layer**: Append-only, raw data with full ingestion metadata\n3. **Silver Layer**: Cleansed data with SCD Type 2 for historical tracking\n4. **Gold Layer**: Star schema with dimensions, facts, and pre-aggregated summaries\n5. **Data Lineage**: Track every transformation for auditability and debugging\n6. **Incremental Processing**: Use watermarks to process only new/changed data\n7. **Quality Gates**: Validate data at each layer before promotion\n\n### Integration with Existing Skill\n\nThis data warehousing pattern integrates with the existing skill:\n\n| Existing Pattern | DWH Application |\n|------------------|-----------------|\n| Schema separation | Extended to bronze/silver/gold |\n| Table API | ETL procedures are the \"API\" to DWH |\n| SECURITY DEFINER | Use for ETL procedures accessing multiple schemas |\n| Naming conventions | Apply l_, in_, io_ prefixes in ETL code |\n| Migration system | Use for DWH schema versioning |\n| Audit logging | Complemented by lineage tracking |\n\n### Recommended Reading Order\n\n1. [Schema Organization](#schema-organization) - Understand the structure\n2. [Bronze Layer](#bronze-layer-raw) - Start with ingestion\n3. [Silver Layer](#silver-layer-cleansed) - Learn SCD Type 2\n4. [Gold Layer](#gold-layer-business) - Build star schemas\n5. [Data Lineage](#data-lineage-tracking) - Add observability\n6. [ETL Orchestration](#etl-orchestration) - Tie it all together\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":74853,"content_sha256":"3eb859483d9bc6f702a237bd4f69949f14a3abe57d89ff7b0101c2bf745a2198"},{"filename":"references/encryption.md","content":"# Encryption Patterns\n\nThis document covers data encryption strategies in PostgreSQL including column-level encryption with pgcrypto, TLS configuration, and sensitive data handling.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Encryption Types](#encryption-types)\n3. [pgcrypto Extension](#pgcrypto-extension)\n4. [Column-Level Encryption](#column-level-encryption)\n5. [Key Management](#key-management)\n6. [TLS/SSL Configuration](#tlsssl-configuration)\n7. [Data Masking](#data-masking)\n8. [Best Practices](#best-practices)\n\n## Overview\n\n### Encryption Strategy Decision Tree\n\n```mermaid\nflowchart TD\n START([Data Security Need]) --> Q1{What layer?}\n\n Q1 -->|\"In transit\"| TLS[\"TLS/SSL\u003cbr/>Client-server encryption\"]\n Q1 -->|\"At rest (disk)\"| Q2{Full disk or selective?}\n Q1 -->|\"In database\"| Q3{All data or specific columns?}\n\n Q2 -->|\"Full disk\"| DISK[\"OS/Storage encryption\u003cbr/>(LUKS, BitLocker)\"]\n Q2 -->|\"Selective\"| TDE[\"Transparent Data Encryption\u003cbr/>(Enterprise feature)\"]\n\n Q3 -->|\"Specific columns\"| COLUMN[\"Column-level encryption\u003cbr/>(pgcrypto)\"]\n Q3 -->|\"Application handles\"| APP[\"Application-level encryption\"]\n\n style TLS fill:#c8e6c9\n style COLUMN fill:#c8e6c9\n```\n\n### Encryption Comparison\n\n| Approach | Protects Against | Performance Impact | Key Location |\n|----------|------------------|-------------------|--------------|\n| TLS | Network sniffing | Low | Certificate store |\n| Disk encryption | Physical theft | Low | OS/Hardware |\n| Column encryption | DB compromise, DBAs | Medium | Application/HSM |\n| Application encryption | All above | High | Application |\n\n## Encryption Types\n\n### Symmetric vs Asymmetric\n\n```sql\n-- Symmetric (AES): Same key encrypts and decrypts\n-- Fast, good for large data\n-- Challenge: Key distribution\n\n-- Asymmetric (RSA): Public key encrypts, private key decrypts\n-- Slower, good for small data or key exchange\n-- Benefit: Public key can be shared\n\n-- Hybrid: Use asymmetric to encrypt symmetric key\n-- Best of both worlds\n```\n\n## pgcrypto Extension\n\n### Installation\n\n```sql\n-- Install pgcrypto extension\nCREATE EXTENSION IF NOT EXISTS pgcrypto;\n\n-- Verify installation\nSELECT * FROM pg_extension WHERE extname = 'pgcrypto';\n```\n\n### Available Functions\n\n```sql\n-- Hashing\ndigest(data, algorithm) -- MD5, SHA1, SHA256, SHA384, SHA512\nhmac(data, key, algorithm) -- Keyed hash\n\n-- Password hashing\ncrypt(password, salt) -- Blowfish-based hashing\ngen_salt(algorithm) -- Generate salt\n\n-- Symmetric encryption\nencrypt(data, key, algorithm) -- AES, Blowfish, etc.\ndecrypt(data, key, algorithm) -- Decrypt\n\n-- Asymmetric encryption (PGP)\npgp_sym_encrypt(data, password)\npgp_sym_decrypt(data, password)\npgp_pub_encrypt(data, key)\npgp_pub_decrypt(data, key)\n\n-- Random data\ngen_random_bytes(count)\ngen_random_uuid()\n```\n\n### Basic Encryption Example\n\n```sql\n-- Symmetric encryption with AES\nSELECT encrypt('sensitive data', 'my-secret-key', 'aes');\n-- Returns: \\x... (bytea)\n\nSELECT convert_from(\n decrypt('\\x...', 'my-secret-key', 'aes'),\n 'UTF8'\n);\n-- Returns: sensitive data\n\n-- PGP symmetric encryption (includes integrity check)\nSELECT pgp_sym_encrypt('sensitive data', 'my-password');\n-- Returns: \\x... (bytea)\n\nSELECT pgp_sym_decrypt('\\x...', 'my-password');\n-- Returns: sensitive data\n```\n\n## Column-Level Encryption\n\n### Table Design with Encrypted Columns\n\n```sql\nCREATE TABLE data.users (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n email text NOT NULL UNIQUE, -- Not encrypted (for lookups)\n email_hash text NOT NULL, -- For searching encrypted email\n\n -- Encrypted sensitive data\n ssn_encrypted bytea,\n phone_encrypted bytea,\n\n -- Non-sensitive data\n name text NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Index on hash for searching\nCREATE INDEX users_email_hash_idx ON data.users(email_hash);\n```\n\n### Encryption Helper Functions\n\n```sql\n-- Encrypt sensitive data\nCREATE FUNCTION private.encrypt_sensitive(in_data text, in_key text)\nRETURNS bytea\nLANGUAGE sql\nIMMUTABLE\nSTRICT\nAS $\n SELECT pgp_sym_encrypt(in_data, in_key, 'cipher-algo=aes256');\n$;\n\n-- Decrypt sensitive data\nCREATE FUNCTION private.decrypt_sensitive(in_data bytea, in_key text)\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nSTRICT\nAS $\n SELECT pgp_sym_decrypt(in_data, in_key);\n$;\n\n-- Generate searchable hash (for encrypted columns)\nCREATE FUNCTION private.hash_for_search(in_data text, in_salt text)\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nSTRICT\nAS $\n SELECT encode(digest(lower(in_data) || in_salt, 'sha256'), 'hex');\n$;\n```\n\n### API Functions with Encryption\n\n```sql\n-- Insert with encryption\nCREATE PROCEDURE api.insert_user(\n in_email text,\n in_ssn text,\n in_phone text,\n in_name text,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_encryption_key text;\n l_hash_salt text;\nBEGIN\n -- Get keys from secure configuration\n l_encryption_key := current_setting('app.encryption_key');\n l_hash_salt := current_setting('app.hash_salt');\n\n INSERT INTO data.users (\n email,\n email_hash,\n ssn_encrypted,\n phone_encrypted,\n name\n ) VALUES (\n in_email,\n private.hash_for_search(in_email, l_hash_salt),\n private.encrypt_sensitive(in_ssn, l_encryption_key),\n private.encrypt_sensitive(in_phone, l_encryption_key),\n in_name\n )\n RETURNING id INTO io_id;\nEND;\n$;\n\n-- Select with decryption\nCREATE FUNCTION api.get_user_details(in_user_id uuid)\nRETURNS TABLE (\n id uuid,\n email text,\n ssn text,\n phone text,\n name text\n)\nLANGUAGE plpgsql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_encryption_key text;\nBEGIN\n l_encryption_key := current_setting('app.encryption_key');\n\n RETURN QUERY\n SELECT\n u.id,\n u.email,\n private.decrypt_sensitive(u.ssn_encrypted, l_encryption_key) AS ssn,\n private.decrypt_sensitive(u.phone_encrypted, l_encryption_key) AS phone,\n u.name\n FROM data.users u\n WHERE u.id = in_user_id;\nEND;\n$;\n```\n\n### Searchable Encryption\n\n```sql\n-- Search by encrypted email using hash\nCREATE FUNCTION api.find_user_by_email(in_email text)\nRETURNS TABLE (\n id uuid,\n email text,\n name text\n)\nLANGUAGE plpgsql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_hash_salt text;\n l_email_hash text;\nBEGIN\n l_hash_salt := current_setting('app.hash_salt');\n l_email_hash := private.hash_for_search(in_email, l_hash_salt);\n\n RETURN QUERY\n SELECT u.id, u.email, u.name\n FROM data.users u\n WHERE u.email_hash = l_email_hash;\nEND;\n$;\n```\n\n## Key Management\n\n### Key Storage Options\n\n```sql\n-- Option 1: PostgreSQL GUC variables (set at connection)\n-- Set in postgresql.conf or per-session\n-- NOT RECOMMENDED for production (visible in logs, pg_settings)\nSET app.encryption_key = 'secret-key';\n\n-- Option 2: Environment variables via file\n-- Create file with restricted permissions\n-- Reference in postgresql.conf:\n-- app.encryption_key = 'include:/path/to/keyfile'\n\n-- Option 3: External secrets manager (recommended)\n-- Application retrieves key and passes to database\n-- Key never stored in database\n```\n\n### Key Rotation Pattern\n\n```sql\n-- Table to track encryption key versions\nCREATE TABLE private.encryption_keys (\n key_id integer PRIMARY KEY,\n key_hash text NOT NULL, -- Hash of key for verification\n created_at timestamptz NOT NULL DEFAULT now(),\n retired_at timestamptz,\n is_active boolean NOT NULL DEFAULT true\n);\n\n-- Add key_version to encrypted tables\nALTER TABLE data.users ADD COLUMN encryption_key_id integer DEFAULT 1;\n\n-- Re-encryption procedure\nCREATE PROCEDURE private.rotate_encryption_key(\n in_old_key text,\n in_new_key text,\n in_new_key_id integer\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_user record;\nBEGIN\n FOR l_user IN\n SELECT id, ssn_encrypted, phone_encrypted\n FROM data.users\n WHERE encryption_key_id != in_new_key_id\n FOR UPDATE\n LOOP\n UPDATE data.users\n SET\n ssn_encrypted = private.encrypt_sensitive(\n private.decrypt_sensitive(l_user.ssn_encrypted, in_old_key),\n in_new_key\n ),\n phone_encrypted = private.encrypt_sensitive(\n private.decrypt_sensitive(l_user.phone_encrypted, in_old_key),\n in_new_key\n ),\n encryption_key_id = in_new_key_id\n WHERE id = l_user.id;\n\n -- Commit in batches to avoid long transaction\n IF l_user.id::text LIKE '%0' THEN\n COMMIT;\n END IF;\n END LOOP;\nEND;\n$;\n```\n\n### Hardware Security Module (HSM) Integration\n\n```sql\n-- For HSM integration, encryption/decryption happens outside PostgreSQL\n-- Database stores only encrypted blobs\n\n-- Example with external encryption (pseudocode)\n-- 1. Application encrypts data using HSM\n-- 2. Encrypted blob sent to PostgreSQL\n-- 3. Retrieval: encrypted blob returned to application\n-- 4. Application decrypts using HSM\n\nCREATE TABLE data.hsm_encrypted_data (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n key_id text NOT NULL, -- HSM key identifier\n encrypted_data bytea NOT NULL, -- HSM-encrypted blob\n created_at timestamptz NOT NULL DEFAULT now()\n);\n```\n\n## TLS/SSL Configuration\n\n### Server Configuration\n\n```bash\n# postgresql.conf\nssl = on\nssl_cert_file = '/path/to/server.crt'\nssl_key_file = '/path/to/server.key'\nssl_ca_file = '/path/to/ca.crt' # For client cert verification\nssl_crl_file = '/path/to/crl.pem' # Certificate revocation list\n\n# Minimum TLS version\nssl_min_protocol_version = 'TLSv1.2'\n\n# Cipher suites (strong ciphers only)\nssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL'\n\n# Prefer server cipher order\nssl_prefer_server_ciphers = on\n```\n\n### Require SSL in pg_hba.conf\n\n```bash\n# Require SSL for all remote connections\nhostssl all all 0.0.0.0/0 scram-sha-256\nhostssl all all ::0/0 scram-sha-256\n\n# Allow non-SSL only for local\nlocal all all peer\nhost all all 127.0.0.1/32 scram-sha-256\n```\n\n### Client Connection with SSL\n\n```bash\n# Connection string\npostgresql://user:pass@host:5432/db?sslmode=verify-full&sslrootcert=/path/to/ca.crt\n\n# psql\npsql \"host=server dbname=mydb sslmode=verify-full sslrootcert=/path/to/ca.crt\"\n```\n\n### SSL Modes\n\n| Mode | Encryption | Certificate Verification |\n|------|------------|-------------------------|\n| disable | No | No |\n| allow | If available | No |\n| prefer | If available | No |\n| require | Yes | No |\n| verify-ca | Yes | CA only |\n| verify-full | Yes | CA + hostname |\n\n### Verify SSL Connection\n\n```sql\n-- Check if connection is encrypted\nSELECT ssl, version, cipher FROM pg_stat_ssl WHERE pid = pg_backend_pid();\n\n-- All active SSL connections\nSELECT\n usename,\n client_addr,\n ssl,\n version,\n cipher,\n bits\nFROM pg_stat_ssl\nJOIN pg_stat_activity ON pg_stat_ssl.pid = pg_stat_activity.pid;\n```\n\n## Data Masking\n\n### Dynamic Data Masking\n\n```sql\n-- Masking functions\nCREATE FUNCTION private.mask_email(in_email text)\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nAS $\n SELECT\n substring(in_email from 1 for 2) ||\n repeat('*', position('@' in in_email) - 3) ||\n substring(in_email from position('@' in in_email));\n -- [email protected] → jo*****@example.com\n$;\n\nCREATE FUNCTION private.mask_phone(in_phone text)\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nAS $\n SELECT\n regexp_replace(in_phone, '(\\d{3})\\d{4}(\\d{4})', '\\1****\\2');\n -- 1234567890 → 123****7890\n$;\n\nCREATE FUNCTION private.mask_ssn(in_ssn text)\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nAS $\n SELECT 'XXX-XX-' || right(in_ssn, 4);\n -- 123-45-6789 → XXX-XX-6789\n$;\n\nCREATE FUNCTION private.mask_credit_card(in_cc text)\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nAS $\n SELECT repeat('*', length(in_cc) - 4) || right(in_cc, 4);\n -- 4111111111111111 → ************1111\n$;\n```\n\n### Role-Based Data Access\n\n```sql\n-- Create view with conditional masking\nCREATE VIEW api.v_users AS\nSELECT\n id,\n CASE\n WHEN current_setting('app.role', true) = 'admin'\n THEN email\n ELSE private.mask_email(email)\n END AS email,\n CASE\n WHEN current_setting('app.role', true) = 'admin'\n THEN private.decrypt_sensitive(ssn_encrypted, current_setting('app.encryption_key'))\n ELSE private.mask_ssn(\n private.decrypt_sensitive(ssn_encrypted, current_setting('app.encryption_key'))\n )\n END AS ssn,\n name,\n created_at\nFROM data.users;\n\n-- Set role before querying\nSET app.role = 'admin'; -- Full access\nSET app.role = 'user'; -- Masked data\n```\n\n### Static Data Masking (for non-production)\n\n```sql\n-- Procedure to mask data for dev/test environments\nCREATE PROCEDURE private.mask_for_nonprod()\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Mask emails\n UPDATE data.users\n SET email = 'user' || id || '@example.com';\n\n -- Randomize SSN (keep format)\n UPDATE data.users\n SET ssn_encrypted = private.encrypt_sensitive(\n lpad((random() * 999)::integer::text, 3, '0') || '-' ||\n lpad((random() * 99)::integer::text, 2, '0') || '-' ||\n lpad((random() * 9999)::integer::text, 4, '0'),\n current_setting('app.encryption_key')\n );\n\n -- Randomize phone numbers\n UPDATE data.users\n SET phone_encrypted = private.encrypt_sensitive(\n lpad((random() * 9999999999)::bigint::text, 10, '0'),\n current_setting('app.encryption_key')\n );\nEND;\n$;\n```\n\n## Best Practices\n\n### Password Storage\n\n```sql\n-- NEVER store plain passwords\n-- Use bcrypt via crypt()\n\nCREATE FUNCTION private.hash_password(in_password text)\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nAS $\n SELECT crypt(in_password, gen_salt('bf', 10)); -- Blowfish, cost 10\n$;\n\nCREATE FUNCTION private.verify_password(in_password text, in_hash text)\nRETURNS boolean\nLANGUAGE sql\nIMMUTABLE\nAS $\n SELECT crypt(in_password, in_hash) = in_hash;\n$;\n\n-- Usage\nINSERT INTO data.users (email, password_hash)\nVALUES ('[email protected]', private.hash_password('secret123'));\n\n-- Verify\nSELECT private.verify_password('secret123', password_hash)\nFROM data.users WHERE email = '[email protected]';\n```\n\n### Audit Encrypted Data Access\n\n```sql\n-- Log access to sensitive data\nCREATE TABLE data.sensitive_data_access_log (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n table_name text NOT NULL,\n record_id uuid NOT NULL,\n accessed_by text NOT NULL DEFAULT current_user,\n accessed_at timestamptz NOT NULL DEFAULT now(),\n access_type text NOT NULL, -- 'view', 'decrypt'\n client_ip inet DEFAULT inet_client_addr()\n);\n\n-- Wrapper function that logs access\nCREATE FUNCTION api.get_user_ssn(in_user_id uuid)\nRETURNS text\nLANGUAGE plpgsql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_ssn text;\nBEGIN\n -- Log the access\n INSERT INTO data.sensitive_data_access_log (table_name, record_id, access_type)\n VALUES ('users', in_user_id, 'decrypt');\n\n -- Return decrypted data\n SELECT private.decrypt_sensitive(ssn_encrypted, current_setting('app.encryption_key'))\n INTO l_ssn\n FROM data.users WHERE id = in_user_id;\n\n RETURN l_ssn;\nEND;\n$;\n```\n\n### Security Checklist\n\n```markdown\n## Encryption Security Checklist\n\n### Key Management\n- [ ] Keys stored outside database\n- [ ] Keys rotated periodically\n- [ ] Key access logged\n- [ ] Separate keys per environment\n- [ ] Key backup procedure documented\n\n### Data Protection\n- [ ] PII identified and encrypted\n- [ ] Encryption algorithms current (AES-256)\n- [ ] Hash salts unique per record/type\n- [ ] Sensitive data masked in logs\n\n### Network Security\n- [ ] TLS enabled and enforced\n- [ ] TLS 1.2+ only\n- [ ] Certificate validation enabled\n- [ ] Certificates rotated before expiry\n\n### Access Control\n- [ ] Decryption limited to necessary roles\n- [ ] Audit logging for sensitive data access\n- [ ] Non-production environments use masked data\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":16186,"content_sha256":"3ad4e93b6bd04d2ee0ae8baa6a46be2436cc31722fb18ebcecd7018df6477ce4"},{"filename":"references/event-sourcing.md","content":"# Event Sourcing & CQRS Patterns\n\nThis document covers event sourcing implementation in PostgreSQL, including event store design, projections, snapshotting, and CQRS patterns.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Event Store Design](#event-store-design)\n3. [Storing Events](#storing-events)\n4. [Reading Events](#reading-events)\n5. [Projections](#projections)\n6. [Snapshotting](#snapshotting)\n7. [CQRS Implementation](#cqrs-implementation)\n8. [Event Versioning](#event-versioning)\n9. [Best Practices](#best-practices)\n\n## Overview\n\n### Event Sourcing Concepts\n\n```mermaid\nflowchart TB\n subgraph WRITE[\"Write Side\"]\n CMD[\"Command\"] --> AGG[\"Aggregate\"]\n AGG --> EVENTS[\"Events\"]\n EVENTS --> STORE[(\"Event Store\")]\n end\n\n subgraph READ[\"Read Side (CQRS)\"]\n STORE --> PROJ[\"Projections\"]\n PROJ --> READ_MODEL[(\"Read Models\")]\n READ_MODEL --> QUERY[\"Queries\"]\n end\n\n style STORE fill:#c8e6c9\n style READ_MODEL fill:#bbdefb\n```\n\n### When to Use Event Sourcing\n\n| Use Case | Event Sourcing | Traditional CRUD |\n|----------|---------------|------------------|\n| Audit requirements | ✅ Built-in | Requires extra work |\n| Time travel/replay | ✅ Natural | Not possible |\n| Complex domain | ✅ Good fit | Can work |\n| Simple CRUD | ❌ Overkill | ✅ Better |\n| High write volume | ⚠️ Consider | ✅ Simpler |\n| Debugging/forensics | ✅ Excellent | Limited |\n\n## Event Store Design\n\n### Core Event Store Table\n\n```sql\n-- Event store table\nCREATE TABLE data.events (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n stream_type text NOT NULL, -- e.g., 'Order', 'Customer'\n stream_id uuid NOT NULL, -- Aggregate ID\n version integer NOT NULL, -- Sequence within stream\n event_type text NOT NULL, -- e.g., 'OrderCreated'\n event_data jsonb NOT NULL, -- Event payload\n metadata jsonb NOT NULL DEFAULT '{}', -- Correlation ID, user, etc.\n created_at timestamptz NOT NULL DEFAULT now(),\n\n -- Ensure optimistic concurrency\n UNIQUE (stream_type, stream_id, version)\n);\n\n-- Indexes for common access patterns\nCREATE INDEX events_stream_idx ON data.events(stream_type, stream_id, version);\nCREATE INDEX events_type_idx ON data.events(event_type);\nCREATE INDEX events_created_idx ON data.events(created_at);\n\n-- For global ordering (useful for projections)\nCREATE INDEX events_id_idx ON data.events(id);\n\nCOMMENT ON TABLE data.events IS 'Append-only event store';\n```\n\n### Event Metadata\n\n```sql\n-- Standard metadata fields\n-- {\n-- \"correlation_id\": \"uuid\", -- Links related events\n-- \"causation_id\": \"uuid\", -- Event that caused this event\n-- \"user_id\": \"uuid\", -- Who triggered it\n-- \"timestamp\": \"iso8601\", -- When it was created\n-- \"version\": 1, -- Event schema version\n-- \"source\": \"api/import/saga\" -- Origin of event\n-- }\n```\n\n### Stream Metadata Table (Optional)\n\n```sql\n-- Track stream state (for validation)\nCREATE TABLE data.event_streams (\n stream_type text NOT NULL,\n stream_id uuid NOT NULL,\n current_version integer NOT NULL DEFAULT 0,\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (stream_type, stream_id)\n);\n\n-- Trigger to update version\nCREATE FUNCTION private.update_stream_version()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nBEGIN\n INSERT INTO data.event_streams (stream_type, stream_id, current_version)\n VALUES (NEW.stream_type, NEW.stream_id, NEW.version)\n ON CONFLICT (stream_type, stream_id) DO UPDATE\n SET current_version = EXCLUDED.current_version,\n updated_at = now();\n RETURN NEW;\nEND;\n$;\n\nCREATE TRIGGER events_stream_version_trg\n AFTER INSERT ON data.events\n FOR EACH ROW\n EXECUTE FUNCTION private.update_stream_version();\n```\n\n## Storing Events\n\n### Append Events Procedure\n\n```sql\nCREATE PROCEDURE api.append_events(\n in_stream_type text,\n in_stream_id uuid,\n in_expected_version integer,\n in_events jsonb, -- Array of {event_type, event_data, metadata}\n INOUT io_new_version integer DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_current_version integer;\n l_event jsonb;\n l_version integer;\nBEGIN\n -- Get current version (with lock)\n SELECT COALESCE(MAX(version), 0) INTO l_current_version\n FROM data.events\n WHERE stream_type = in_stream_type AND stream_id = in_stream_id\n FOR UPDATE;\n\n -- Optimistic concurrency check\n IF l_current_version != in_expected_version THEN\n RAISE EXCEPTION 'Concurrency conflict: expected version %, but found %',\n in_expected_version, l_current_version\n USING ERRCODE = 'P0002';\n END IF;\n\n -- Append each event\n l_version := l_current_version;\n FOR l_event IN SELECT * FROM jsonb_array_elements(in_events)\n LOOP\n l_version := l_version + 1;\n\n INSERT INTO data.events (\n stream_type, stream_id, version, event_type, event_data, metadata\n ) VALUES (\n in_stream_type,\n in_stream_id,\n l_version,\n l_event->>'event_type',\n l_event->'event_data',\n COALESCE(l_event->'metadata', '{}')\n );\n END LOOP;\n\n io_new_version := l_version;\nEND;\n$;\n```\n\n### Usage Example\n\n```sql\n-- Create a new order\nCALL api.append_events(\n in_stream_type := 'Order',\n in_stream_id := 'order-uuid',\n in_expected_version := 0, -- New stream\n in_events := '[\n {\n \"event_type\": \"OrderCreated\",\n \"event_data\": {\n \"customer_id\": \"customer-uuid\",\n \"items\": [{\"product_id\": \"prod-1\", \"quantity\": 2}]\n },\n \"metadata\": {\"user_id\": \"user-uuid\", \"correlation_id\": \"request-uuid\"}\n }\n ]'::jsonb\n);\n\n-- Add items to existing order\nCALL api.append_events(\n in_stream_type := 'Order',\n in_stream_id := 'order-uuid',\n in_expected_version := 1, -- After OrderCreated\n in_events := '[\n {\n \"event_type\": \"ItemAdded\",\n \"event_data\": {\"product_id\": \"prod-2\", \"quantity\": 1}\n }\n ]'::jsonb\n);\n```\n\n## Reading Events\n\n### Get Events for Stream\n\n```sql\nCREATE FUNCTION api.get_events(\n in_stream_type text,\n in_stream_id uuid,\n in_from_version integer DEFAULT 0\n)\nRETURNS TABLE (\n id uuid,\n version integer,\n event_type text,\n event_data jsonb,\n metadata jsonb,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, version, event_type, event_data, metadata, created_at\n FROM data.events\n WHERE stream_type = in_stream_type\n AND stream_id = in_stream_id\n AND version > in_from_version\n ORDER BY version;\n$;\n```\n\n### Get Events by Type (for projections)\n\n```sql\nCREATE FUNCTION api.get_events_by_type(\n in_event_types text[],\n in_after_id uuid DEFAULT NULL,\n in_limit integer DEFAULT 1000\n)\nRETURNS TABLE (\n id uuid,\n stream_type text,\n stream_id uuid,\n version integer,\n event_type text,\n event_data jsonb,\n metadata jsonb,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, stream_type, stream_id, version, event_type, event_data, metadata, created_at\n FROM data.events\n WHERE event_type = ANY(in_event_types)\n AND (in_after_id IS NULL OR id > in_after_id)\n ORDER BY id\n LIMIT in_limit;\n$;\n```\n\n### Get All Events (for global projection)\n\n```sql\nCREATE FUNCTION api.get_all_events(\n in_after_id uuid DEFAULT NULL,\n in_limit integer DEFAULT 1000\n)\nRETURNS TABLE (\n id uuid,\n stream_type text,\n stream_id uuid,\n version integer,\n event_type text,\n event_data jsonb,\n metadata jsonb,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, stream_type, stream_id, version, event_type, event_data, metadata, created_at\n FROM data.events\n WHERE in_after_id IS NULL OR id > in_after_id\n ORDER BY id\n LIMIT in_limit;\n$;\n```\n\n## Projections\n\n### Projection Tracking Table\n\n```sql\n-- Track projection positions\nCREATE TABLE data.projection_checkpoints (\n projection_name text PRIMARY KEY,\n last_event_id uuid,\n last_processed timestamptz NOT NULL DEFAULT now(),\n status text NOT NULL DEFAULT 'running',\n error_message text\n);\n```\n\n### Example: Order Summary Projection\n\n```sql\n-- Read model table\nCREATE TABLE data.order_summaries (\n order_id uuid PRIMARY KEY,\n customer_id uuid NOT NULL,\n status text NOT NULL,\n total_items integer NOT NULL DEFAULT 0,\n total_amount numeric(15,2) NOT NULL DEFAULT 0,\n created_at timestamptz NOT NULL,\n updated_at timestamptz NOT NULL\n);\n\n-- Projection handler\nCREATE PROCEDURE private.project_order_events(in_batch_size integer DEFAULT 100)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_checkpoint uuid;\n l_event record;\n l_processed integer := 0;\nBEGIN\n -- Get last checkpoint\n SELECT last_event_id INTO l_checkpoint\n FROM data.projection_checkpoints\n WHERE projection_name = 'order_summaries';\n\n -- Process events\n FOR l_event IN\n SELECT * FROM data.events\n WHERE stream_type = 'Order'\n AND (l_checkpoint IS NULL OR id > l_checkpoint)\n ORDER BY id\n LIMIT in_batch_size\n LOOP\n -- Apply event to read model\n CASE l_event.event_type\n WHEN 'OrderCreated' THEN\n INSERT INTO data.order_summaries (\n order_id, customer_id, status, created_at, updated_at\n ) VALUES (\n l_event.stream_id,\n (l_event.event_data->>'customer_id')::uuid,\n 'created',\n l_event.created_at,\n l_event.created_at\n );\n\n WHEN 'ItemAdded' THEN\n UPDATE data.order_summaries\n SET total_items = total_items + (l_event.event_data->>'quantity')::integer,\n updated_at = l_event.created_at\n WHERE order_id = l_event.stream_id;\n\n WHEN 'OrderSubmitted' THEN\n UPDATE data.order_summaries\n SET status = 'submitted',\n updated_at = l_event.created_at\n WHERE order_id = l_event.stream_id;\n\n WHEN 'OrderCancelled' THEN\n UPDATE data.order_summaries\n SET status = 'cancelled',\n updated_at = l_event.created_at\n WHERE order_id = l_event.stream_id;\n END CASE;\n\n l_checkpoint := l_event.id;\n l_processed := l_processed + 1;\n END LOOP;\n\n -- Update checkpoint\n IF l_processed > 0 THEN\n INSERT INTO data.projection_checkpoints (projection_name, last_event_id)\n VALUES ('order_summaries', l_checkpoint)\n ON CONFLICT (projection_name) DO UPDATE\n SET last_event_id = EXCLUDED.last_event_id,\n last_processed = now();\n END IF;\nEND;\n$;\n\n-- Schedule projection updates\nSELECT cron.schedule(\n 'order-summaries-projection',\n '*/5 * * * * *', -- Every 5 seconds\n $CALL private.project_order_events(100)$\n);\n```\n\n### Rebuild Projection\n\n```sql\nCREATE PROCEDURE api.rebuild_projection(in_projection_name text)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n CASE in_projection_name\n WHEN 'order_summaries' THEN\n -- Clear read model\n TRUNCATE data.order_summaries;\n\n -- Reset checkpoint\n DELETE FROM data.projection_checkpoints\n WHERE projection_name = 'order_summaries';\n\n -- Reprocess all events\n LOOP\n CALL private.project_order_events(1000);\n EXIT WHEN NOT EXISTS (\n SELECT 1 FROM data.events e\n LEFT JOIN data.projection_checkpoints p ON p.projection_name = 'order_summaries'\n WHERE e.stream_type = 'Order'\n AND (p.last_event_id IS NULL OR e.id > p.last_event_id)\n LIMIT 1\n );\n END LOOP;\n\n ELSE\n RAISE EXCEPTION 'Unknown projection: %', in_projection_name;\n END CASE;\nEND;\n$;\n```\n\n## Snapshotting\n\n### Snapshot Table\n\n```sql\nCREATE TABLE data.snapshots (\n stream_type text NOT NULL,\n stream_id uuid NOT NULL,\n version integer NOT NULL,\n state jsonb NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (stream_type, stream_id)\n);\n\nCREATE INDEX snapshots_version_idx ON data.snapshots(stream_type, stream_id, version);\n```\n\n### Save Snapshot\n\n```sql\nCREATE PROCEDURE private.save_snapshot(\n in_stream_type text,\n in_stream_id uuid,\n in_version integer,\n in_state jsonb\n)\nLANGUAGE sql\nAS $\n INSERT INTO data.snapshots (stream_type, stream_id, version, state)\n VALUES (in_stream_type, in_stream_id, in_version, in_state)\n ON CONFLICT (stream_type, stream_id) DO UPDATE\n SET version = EXCLUDED.version,\n state = EXCLUDED.state,\n created_at = now();\n$;\n```\n\n### Load Aggregate with Snapshot\n\n```sql\nCREATE FUNCTION api.get_aggregate_state(\n in_stream_type text,\n in_stream_id uuid\n)\nRETURNS TABLE (\n state jsonb,\n version integer,\n pending_events jsonb\n)\nLANGUAGE plpgsql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_snapshot record;\n l_from_version integer := 0;\n l_state jsonb := '{}';\nBEGIN\n -- Try to load snapshot\n SELECT s.version, s.state INTO l_snapshot\n FROM data.snapshots s\n WHERE s.stream_type = in_stream_type AND s.stream_id = in_stream_id;\n\n IF FOUND THEN\n l_from_version := l_snapshot.version;\n l_state := l_snapshot.state;\n END IF;\n\n -- Get events since snapshot\n RETURN QUERY\n SELECT\n l_state AS state,\n l_from_version AS version,\n COALESCE(jsonb_agg(jsonb_build_object(\n 'version', e.version,\n 'event_type', e.event_type,\n 'event_data', e.event_data\n ) ORDER BY e.version), '[]'::jsonb) AS pending_events\n FROM data.events e\n WHERE e.stream_type = in_stream_type\n AND e.stream_id = in_stream_id\n AND e.version > l_from_version;\nEND;\n$;\n```\n\n### Automatic Snapshotting\n\n```sql\n-- Trigger to create snapshot every N events\nCREATE FUNCTION private.maybe_create_snapshot()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nDECLARE\n co_snapshot_interval constant integer := 100;\nBEGIN\n -- Snapshot every 100 events\n IF NEW.version % co_snapshot_interval = 0 THEN\n -- Application should compute state and call save_snapshot\n -- This is a placeholder - actual state computation depends on domain\n RAISE NOTICE 'Consider creating snapshot for %/% at version %',\n NEW.stream_type, NEW.stream_id, NEW.version;\n END IF;\n RETURN NEW;\nEND;\n$;\n\nCREATE TRIGGER events_snapshot_trg\n AFTER INSERT ON data.events\n FOR EACH ROW\n EXECUTE FUNCTION private.maybe_create_snapshot();\n```\n\n## CQRS Implementation\n\n### Command Handler Example\n\n```sql\n-- Command: Create Order\nCREATE PROCEDURE api.create_order(\n in_customer_id uuid,\n in_items jsonb, -- [{product_id, quantity, price}]\n INOUT io_order_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_order_id uuid := uuidv7();\n l_total numeric;\nBEGIN\n -- Validate\n IF NOT EXISTS (SELECT 1 FROM data.customers WHERE id = in_customer_id) THEN\n RAISE EXCEPTION 'Customer not found: %', in_customer_id\n USING ERRCODE = 'P0002';\n END IF;\n\n -- Calculate total\n SELECT SUM((item->>'price')::numeric * (item->>'quantity')::integer)\n INTO l_total\n FROM jsonb_array_elements(in_items) AS item;\n\n -- Append event\n CALL api.append_events(\n in_stream_type := 'Order',\n in_stream_id := l_order_id,\n in_expected_version := 0,\n in_events := jsonb_build_array(\n jsonb_build_object(\n 'event_type', 'OrderCreated',\n 'event_data', jsonb_build_object(\n 'customer_id', in_customer_id,\n 'items', in_items,\n 'total', l_total\n )\n )\n )\n );\n\n io_order_id := l_order_id;\nEND;\n$;\n```\n\n### Query from Read Model\n\n```sql\n-- Query: Get customer orders\nCREATE FUNCTION api.get_customer_orders(\n in_customer_id uuid,\n in_status text DEFAULT NULL,\n in_limit integer DEFAULT 20\n)\nRETURNS TABLE (\n order_id uuid,\n status text,\n total_items integer,\n total_amount numeric,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT order_id, status, total_items, total_amount, created_at\n FROM data.order_summaries\n WHERE customer_id = in_customer_id\n AND (in_status IS NULL OR status = in_status)\n ORDER BY created_at DESC\n LIMIT in_limit;\n$;\n```\n\n## Event Versioning\n\n### Schema Evolution\n\n```sql\n-- Event type registry\nCREATE TABLE data.event_schemas (\n event_type text NOT NULL,\n version integer NOT NULL,\n schema jsonb NOT NULL, -- JSON Schema\n created_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (event_type, version)\n);\n\n-- Upcaster function example\nCREATE FUNCTION private.upcast_event(\n in_event_type text,\n in_event_data jsonb,\n in_from_version integer,\n in_to_version integer\n)\nRETURNS jsonb\nLANGUAGE plpgsql\nIMMUTABLE\nAS $\nBEGIN\n IF in_event_type = 'OrderCreated' THEN\n -- V1 -> V2: Add shipping_address field\n IF in_from_version = 1 AND in_to_version = 2 THEN\n RETURN in_event_data || '{\"shipping_address\": null}'::jsonb;\n END IF;\n END IF;\n\n RETURN in_event_data;\nEND;\n$;\n```\n\n## Best Practices\n\n### Event Design Guidelines\n\n```sql\n-- Good event names (past tense, domain language)\n-- ✅ OrderCreated, ItemAdded, PaymentReceived\n-- ❌ CreateOrder, AddItem, ReceivePayment\n\n-- Include enough context\n-- ✅ {\"product_id\": \"...\", \"quantity\": 2, \"price\": 10.00, \"product_name\": \"Widget\"}\n-- ❌ {\"product_id\": \"...\"} -- Missing context for projections\n```\n\n### Idempotency\n\n```sql\n-- Use idempotency keys in metadata\nCREATE PROCEDURE api.append_events_idempotent(\n in_stream_type text,\n in_stream_id uuid,\n in_expected_version integer,\n in_events jsonb,\n in_idempotency_key uuid\n)\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Check if already processed\n IF EXISTS (\n SELECT 1 FROM data.events\n WHERE metadata->>'idempotency_key' = in_idempotency_key::text\n ) THEN\n RAISE NOTICE 'Event already processed: %', in_idempotency_key;\n RETURN;\n END IF;\n\n -- Add idempotency key to all events\n CALL api.append_events(\n in_stream_type,\n in_stream_id,\n in_expected_version,\n (SELECT jsonb_agg(\n event || jsonb_build_object('metadata',\n COALESCE(event->'metadata', '{}'::jsonb) ||\n jsonb_build_object('idempotency_key', in_idempotency_key)\n )\n ) FROM jsonb_array_elements(in_events) AS event)\n );\nEND;\n$;\n```\n\n### Partitioning Events\n\n```sql\n-- Partition by month for large event stores\nCREATE TABLE data.events (\n id uuid NOT NULL DEFAULT uuidv7(),\n stream_type text NOT NULL,\n stream_id uuid NOT NULL,\n version integer NOT NULL,\n event_type text NOT NULL,\n event_data jsonb NOT NULL,\n metadata jsonb NOT NULL DEFAULT '{}',\n created_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (id, created_at)\n) PARTITION BY RANGE (created_at);\n\n-- Create monthly partitions\nCREATE TABLE data.events_2024_01 PARTITION OF data.events\n FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":20197,"content_sha256":"d180d2c316371dfee3864feb8c5828144ee18fea5ada1d337710aecf5f304ead"},{"filename":"references/full-text-search.md","content":"# Full-Text Search Patterns\n\nThis document covers PostgreSQL's built-in full-text search capabilities, including tsvector/tsquery design, indexing strategies, ranking, and multi-language support.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Core Concepts](#core-concepts)\n3. [Schema Design for FTS](#schema-design-for-fts)\n4. [Indexing Strategies](#indexing-strategies)\n5. [Query Patterns](#query-patterns)\n6. [Search Ranking](#search-ranking)\n7. [Multi-Language Support](#multi-language-support)\n8. [Advanced Patterns](#advanced-patterns)\n9. [Performance Optimization](#performance-optimization)\n\n## Overview\n\n### When to Use PostgreSQL FTS\n\n| Use Case | PostgreSQL FTS | External Search (Elasticsearch) |\n|----------|----------------|--------------------------------|\n| Simple text search | ✅ Excellent | Overkill |\n| Blog/CMS content | ✅ Good | Good |\n| Product search | ✅ Good | Better for facets |\n| Log analysis | ⚠️ Limited | ✅ Better |\n| Real-time search | ✅ Good | ✅ Good |\n| Fuzzy matching | ⚠️ pg_trgm addon | ✅ Built-in |\n| Multi-language | ✅ Good | ✅ Better |\n| Operational complexity | ✅ None (built-in) | ❌ High |\n\n### FTS vs LIKE/ILIKE\n\n```sql\n-- BAD: LIKE with wildcards - cannot use B-tree index\nSELECT * FROM data.articles WHERE title ILIKE '%postgresql%';\n\n-- GOOD: Full-text search - uses GIN index\nSELECT * FROM data.articles\nWHERE to_tsvector('english', title) @@ to_tsquery('english', 'postgresql');\n```\n\n## Core Concepts\n\n### tsvector - Document Representation\n\n```sql\n-- tsvector: normalized document representation\nSELECT to_tsvector('english', 'The quick brown foxes jumped over the lazy dogs');\n-- Result: 'brown':3 'dog':9 'fox':4 'jump':5 'lazi':8 'quick':2\n\n-- Features:\n-- - Lowercased\n-- - Stop words removed ('the', 'over')\n-- - Stemmed ('foxes' → 'fox', 'jumped' → 'jump')\n-- - Position information stored\n```\n\n### tsquery - Search Query\n\n```sql\n-- Basic query\nSELECT to_tsquery('english', 'quick & brown');\n-- Result: 'quick' & 'brown'\n\n-- Query operators:\n-- & AND\n-- | OR\n-- ! NOT\n-- \u003c-> FOLLOWED BY (phrase)\n-- \u003cN> FOLLOWED BY within N words\n\n-- Phrase search\nSELECT to_tsquery('english', 'quick \u003c-> brown');\n-- Result: 'quick' \u003c-> 'brown' (must be adjacent)\n\n-- Proximity search\nSELECT to_tsquery('english', 'quick \u003c2> fox');\n-- Result: 'quick' \u003c2> 'fox' (within 2 words)\n```\n\n### Match Operator (@@)\n\n```sql\n-- Check if document matches query\nSELECT to_tsvector('english', 'The quick brown fox')\n @@ to_tsquery('english', 'quick & fox');\n-- Result: true\n\nSELECT to_tsvector('english', 'The quick brown fox')\n @@ to_tsquery('english', 'quick & cat');\n-- Result: false\n```\n\n## Schema Design for FTS\n\n### Option 1: Computed Column (Recommended)\n\n```sql\n-- Store precomputed tsvector as generated column\nCREATE TABLE data.articles (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n title text NOT NULL,\n body text NOT NULL,\n author_id uuid NOT NULL REFERENCES data.users(id),\n\n -- Generated tsvector column (stored)\n search_vector tsvector GENERATED ALWAYS AS (\n setweight(to_tsvector('english', coalesce(title, '')), 'A') ||\n setweight(to_tsvector('english', coalesce(body, '')), 'B')\n ) STORED,\n\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- GIN index on the tsvector column\nCREATE INDEX articles_search_idx ON data.articles USING gin(search_vector);\n```\n\n### Option 2: Trigger-Maintained Column\n\n```sql\n-- For more complex logic or PostgreSQL \u003c 12\nCREATE TABLE data.products (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n description text,\n category text NOT NULL,\n tags text[],\n search_vector tsvector,\n\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Trigger to maintain search vector\nCREATE FUNCTION private.products_search_vector_trigger()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nBEGIN\n NEW.search_vector :=\n setweight(to_tsvector('english', coalesce(NEW.name, '')), 'A') ||\n setweight(to_tsvector('english', coalesce(NEW.description, '')), 'B') ||\n setweight(to_tsvector('english', coalesce(NEW.category, '')), 'C') ||\n setweight(to_tsvector('english', coalesce(array_to_string(NEW.tags, ' '), '')), 'C');\n RETURN NEW;\nEND;\n$;\n\nCREATE TRIGGER products_search_vector_trg\n BEFORE INSERT OR UPDATE OF name, description, category, tags\n ON data.products\n FOR EACH ROW\n EXECUTE FUNCTION private.products_search_vector_trigger();\n\nCREATE INDEX products_search_idx ON data.products USING gin(search_vector);\n```\n\n### Option 3: On-the-Fly (Simple Cases)\n\n```sql\n-- No stored vector - compute at query time\n-- Only suitable for small tables or infrequent searches\nCREATE TABLE data.notes (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n content text NOT NULL,\n user_id uuid NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Expression index (computed at index time, not stored in table)\nCREATE INDEX notes_content_search_idx\n ON data.notes USING gin(to_tsvector('english', content));\n\n-- Query must match index expression exactly\nSELECT * FROM data.notes\nWHERE to_tsvector('english', content) @@ to_tsquery('english', 'search & term');\n```\n\n## Indexing Strategies\n\n### GIN Index (Recommended)\n\n```sql\n-- Standard GIN index for full-text search\nCREATE INDEX articles_search_idx ON data.articles USING gin(search_vector);\n\n-- Pros:\n-- - Fast lookups\n-- - Efficient for many unique terms\n-- - Good for read-heavy workloads\n\n-- Cons:\n-- - Slower index updates than GiST\n-- - Larger index size\n```\n\n### GiST Index (Alternative)\n\n```sql\n-- GiST index - faster updates, slower lookups\nCREATE INDEX articles_search_gist_idx ON data.articles USING gist(search_vector);\n\n-- Use when:\n-- - Write-heavy workload\n-- - Smaller document corpus\n-- - Combined with geometric/range queries\n```\n\n### Partial Index for Common Filters\n\n```sql\n-- Index only published articles\nCREATE INDEX articles_search_published_idx\n ON data.articles USING gin(search_vector)\n WHERE status = 'published';\n\n-- Smaller index, faster queries for common case\nSELECT * FROM data.articles\nWHERE search_vector @@ to_tsquery('english', 'postgresql')\n AND status = 'published';\n```\n\n### Multi-Column Index\n\n```sql\n-- Combine FTS with other filters\nCREATE INDEX articles_author_search_idx\n ON data.articles USING gin(author_id, search_vector);\n\n-- Efficient for: WHERE author_id = X AND search_vector @@ query\n```\n\n## Query Patterns\n\n### Basic Search\n\n```sql\n-- Simple word search\nSELECT id, title, ts_headline('english', body, q) AS snippet\nFROM data.articles, to_tsquery('english', 'postgresql') AS q\nWHERE search_vector @@ q\nORDER BY ts_rank(search_vector, q) DESC\nLIMIT 20;\n```\n\n### Phrase Search\n\n```sql\n-- Exact phrase: words must be adjacent\nSELECT * FROM data.articles\nWHERE search_vector @@ phraseto_tsquery('english', 'database optimization');\n\n-- Equivalent to:\nSELECT * FROM data.articles\nWHERE search_vector @@ to_tsquery('english', 'database \u003c-> optimization');\n```\n\n### Boolean Operators\n\n```sql\n-- AND: both terms required\nSELECT * FROM data.articles\nWHERE search_vector @@ to_tsquery('english', 'postgresql & performance');\n\n-- OR: either term\nSELECT * FROM data.articles\nWHERE search_vector @@ to_tsquery('english', 'postgresql | mysql');\n\n-- NOT: exclude term\nSELECT * FROM data.articles\nWHERE search_vector @@ to_tsquery('english', 'postgresql & !mysql');\n\n-- Complex boolean\nSELECT * FROM data.articles\nWHERE search_vector @@ to_tsquery('english', '(postgresql | postgres) & (performance | optimization) & !beginner');\n```\n\n### Prefix Search\n\n```sql\n-- Prefix matching with :*\nSELECT * FROM data.articles\nWHERE search_vector @@ to_tsquery('english', 'postg:*');\n-- Matches: postgresql, postgres, postgis, etc.\n\n-- Combine with other terms\nSELECT * FROM data.articles\nWHERE search_vector @@ to_tsquery('english', 'postg:* & index:*');\n```\n\n### Web-Style Search Input\n\n```sql\n-- Convert user input to tsquery\n-- websearch_to_tsquery handles quotes, -, OR naturally\n\nSELECT * FROM data.articles\nWHERE search_vector @@ websearch_to_tsquery('english', 'postgresql \"query optimization\" -beginner');\n-- Interprets as: postgresql AND \"query optimization\" AND NOT beginner\n\n-- User-friendly search function\nCREATE FUNCTION api.search_articles(in_query text, in_limit integer DEFAULT 20)\nRETURNS TABLE (\n id uuid,\n title text,\n snippet text,\n rank real\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT\n a.id,\n a.title,\n ts_headline('english', a.body, q, 'MaxFragments=2, MaxWords=30') AS snippet,\n ts_rank(a.search_vector, q) AS rank\n FROM data.articles a,\n websearch_to_tsquery('english', in_query) AS q\n WHERE a.search_vector @@ q\n AND a.status = 'published'\n ORDER BY ts_rank(a.search_vector, q) DESC\n LIMIT in_limit;\n$;\n```\n\n### Combining FTS with Other Filters\n\n```sql\n-- FTS with category filter\nSELECT * FROM data.articles\nWHERE search_vector @@ websearch_to_tsquery('english', 'postgresql')\n AND category = 'tutorials'\n AND created_at > now() - interval '1 year'\nORDER BY ts_rank(search_vector, websearch_to_tsquery('english', 'postgresql')) DESC\nLIMIT 20;\n\n-- FTS with pagination (keyset)\nSELECT id, title, ts_rank(search_vector, q) AS rank\nFROM data.articles, websearch_to_tsquery('english', 'postgresql') AS q\nWHERE search_vector @@ q\n AND (ts_rank(search_vector, q), id) \u003c (0.5, 'last-seen-uuid')\nORDER BY ts_rank(search_vector, q) DESC, id DESC\nLIMIT 20;\n```\n\n## Search Ranking\n\n### ts_rank - Basic Ranking\n\n```sql\n-- ts_rank: considers term frequency\nSELECT\n title,\n ts_rank(search_vector, query) AS rank\nFROM data.articles,\n to_tsquery('english', 'postgresql & performance') AS query\nWHERE search_vector @@ query\nORDER BY rank DESC;\n\n-- Normalization options (bitmask):\n-- 0: default\n-- 1: divide by 1 + log(document length)\n-- 2: divide by document length\n-- 4: divide by mean harmonic distance between extents\n-- 8: divide by number of unique words\n-- 16: divide by 1 + log(unique words)\n-- 32: divide by itself + 1\n\nSELECT ts_rank(search_vector, query, 1|4) AS normalized_rank\nFROM data.articles, to_tsquery('english', 'postgresql') AS query\nWHERE search_vector @@ query;\n```\n\n### ts_rank_cd - Cover Density Ranking\n\n```sql\n-- ts_rank_cd: considers proximity of matching terms\n-- Better for phrase-like queries\nSELECT\n title,\n ts_rank_cd(search_vector, query) AS rank\nFROM data.articles,\n to_tsquery('english', 'postgresql & performance & tuning') AS query\nWHERE search_vector @@ query\nORDER BY rank DESC;\n```\n\n### Weighted Ranking\n\n```sql\n-- Weights for A, B, C, D categories (default: {0.1, 0.2, 0.4, 1.0})\nSELECT\n title,\n ts_rank(search_vector, query, '{0.1, 0.2, 0.4, 1.0}') AS rank\nFROM data.articles,\n to_tsquery('english', 'postgresql') AS query\nWHERE search_vector @@ query\nORDER BY rank DESC;\n\n-- Custom weights: prioritize title (A) and category (C)\nSELECT\n title,\n ts_rank(search_vector, query, '{1.0, 0.4, 0.8, 0.1}') AS rank\nFROM data.articles,\n to_tsquery('english', 'postgresql') AS query\nWHERE search_vector @@ query\nORDER BY rank DESC;\n```\n\n### Combined Ranking with Other Factors\n\n```sql\n-- Combine FTS rank with recency\nSELECT\n id,\n title,\n ts_rank(search_vector, query) *\n (1 + 1.0 / (extract(epoch from now() - created_at) / 86400 + 1)) AS combined_rank\nFROM data.articles,\n to_tsquery('english', 'postgresql') AS query\nWHERE search_vector @@ query\nORDER BY combined_rank DESC;\n\n-- Boost by view count\nSELECT\n id,\n title,\n ts_rank(search_vector, query) * (1 + ln(view_count + 1) * 0.1) AS boosted_rank\nFROM data.articles,\n to_tsquery('english', 'postgresql') AS query\nWHERE search_vector @@ query\nORDER BY boosted_rank DESC;\n```\n\n## Multi-Language Support\n\n### Available Dictionaries\n\n```sql\n-- List available text search configurations\nSELECT cfgname FROM pg_ts_config;\n-- simple, danish, dutch, english, finnish, french, german, hungarian,\n-- italian, norwegian, portuguese, romanian, russian, spanish, swedish, turkish\n\n-- Check default configuration\nSHOW default_text_search_config;\n```\n\n### Language-Specific Search\n\n```sql\n-- Create table with language column\nCREATE TABLE data.content (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n title text NOT NULL,\n body text NOT NULL,\n language text NOT NULL DEFAULT 'english',\n search_vector tsvector,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Dynamic language trigger\nCREATE FUNCTION private.content_search_vector_trigger()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_config regconfig;\nBEGIN\n -- Map language to config (with fallback)\n l_config := CASE NEW.language\n WHEN 'en' THEN 'english'::regconfig\n WHEN 'de' THEN 'german'::regconfig\n WHEN 'fr' THEN 'french'::regconfig\n WHEN 'es' THEN 'spanish'::regconfig\n ELSE 'simple'::regconfig\n END;\n\n NEW.search_vector :=\n setweight(to_tsvector(l_config, coalesce(NEW.title, '')), 'A') ||\n setweight(to_tsvector(l_config, coalesce(NEW.body, '')), 'B');\n\n RETURN NEW;\nEND;\n$;\n\nCREATE TRIGGER content_search_trg\n BEFORE INSERT OR UPDATE OF title, body, language\n ON data.content\n FOR EACH ROW\n EXECUTE FUNCTION private.content_search_vector_trigger();\n```\n\n### Multi-Language Search Function\n\n```sql\nCREATE FUNCTION api.search_content(\n in_query text,\n in_language text DEFAULT 'english',\n in_limit integer DEFAULT 20\n)\nRETURNS TABLE (\n id uuid,\n title text,\n snippet text,\n rank real\n)\nLANGUAGE plpgsql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_config regconfig;\nBEGIN\n l_config := CASE in_language\n WHEN 'en' THEN 'english'::regconfig\n WHEN 'de' THEN 'german'::regconfig\n WHEN 'fr' THEN 'french'::regconfig\n WHEN 'es' THEN 'spanish'::regconfig\n ELSE 'simple'::regconfig\n END;\n\n RETURN QUERY\n SELECT\n c.id,\n c.title,\n ts_headline(l_config, c.body, websearch_to_tsquery(l_config, in_query)) AS snippet,\n ts_rank(c.search_vector, websearch_to_tsquery(l_config, in_query)) AS rank\n FROM data.content c\n WHERE c.search_vector @@ websearch_to_tsquery(l_config, in_query)\n AND c.language = in_language\n ORDER BY ts_rank(c.search_vector, websearch_to_tsquery(l_config, in_query)) DESC\n LIMIT in_limit;\nEND;\n$;\n```\n\n### Unaccented Search\n\n```sql\n-- Install unaccent extension\nCREATE EXTENSION IF NOT EXISTS unaccent;\n\n-- Create custom text search configuration with unaccent\nCREATE TEXT SEARCH CONFIGURATION french_unaccent (COPY = french);\nALTER TEXT SEARCH CONFIGURATION french_unaccent\n ALTER MAPPING FOR hword, hword_part, word\n WITH unaccent, french_stem;\n\n-- Now \"café\" matches \"cafe\"\nSELECT to_tsvector('french_unaccent', 'Le café est délicieux');\n-- 'cafe':2 'delicieux':4\n\nSELECT to_tsvector('french_unaccent', 'Le café est délicieux')\n @@ to_tsquery('french_unaccent', 'cafe');\n-- true\n```\n\n## Advanced Patterns\n\n### Highlighting Search Results\n\n```sql\n-- ts_headline: generate highlighted snippets\nSELECT\n title,\n ts_headline(\n 'english',\n body,\n to_tsquery('english', 'postgresql & performance'),\n 'StartSel=\u003cmark>, StopSel=\u003c/mark>, MaxFragments=3, MaxWords=30, MinWords=10'\n ) AS highlighted_snippet\nFROM data.articles\nWHERE search_vector @@ to_tsquery('english', 'postgresql & performance');\n\n-- Options:\n-- StartSel, StopSel: highlight markers\n-- MaxFragments: max number of fragments (0 = entire field)\n-- MaxWords: max words per fragment\n-- MinWords: min words per fragment\n-- ShortWord: words shorter than this are dropped at fragment start/end\n-- HighlightAll: highlight all words even if not in query\n-- FragmentDelimiter: separator between fragments (default: \" ... \")\n```\n\n### Synonym Support\n\n```sql\n-- Create synonym dictionary file: /usr/share/postgresql/tsearch_data/my_synonyms.syn\n-- Contents:\n-- postgres postgresql\n-- psql postgresql\n-- pg postgresql\n-- db database\n\n-- Create text search dictionary\nCREATE TEXT SEARCH DICTIONARY my_synonyms (\n TEMPLATE = synonym,\n SYNONYMS = my_synonyms\n);\n\n-- Create custom configuration\nCREATE TEXT SEARCH CONFIGURATION english_syn (COPY = english);\nALTER TEXT SEARCH CONFIGURATION english_syn\n ALTER MAPPING FOR asciiword, asciihword, hword_asciipart\n WITH my_synonyms, english_stem;\n\n-- Now \"pg\" matches \"postgresql\"\nSELECT to_tsvector('english_syn', 'pg optimization tips');\n-- 'optim':2 'postgresql':1 'tip':3\n```\n\n### Fuzzy Matching with pg_trgm\n\n```sql\n-- For typo-tolerant search, combine FTS with trigram similarity\nCREATE EXTENSION IF NOT EXISTS pg_trgm;\n\n-- Trigram index for fuzzy matching\nCREATE INDEX articles_title_trgm_idx ON data.articles USING gin(title gin_trgm_ops);\n\n-- Combined search: exact FTS + fuzzy fallback\nCREATE FUNCTION api.search_articles_fuzzy(\n in_query text,\n in_limit integer DEFAULT 20\n)\nRETURNS TABLE (\n id uuid,\n title text,\n match_type text,\n score real\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n -- Exact FTS matches\n SELECT id, title, 'exact'::text AS match_type, ts_rank(search_vector, q) AS score\n FROM data.articles, websearch_to_tsquery('english', in_query) AS q\n WHERE search_vector @@ q\n\n UNION ALL\n\n -- Fuzzy title matches (not already in exact results)\n SELECT id, title, 'fuzzy'::text, similarity(title, in_query) AS score\n FROM data.articles\n WHERE similarity(title, in_query) > 0.3\n AND id NOT IN (\n SELECT a.id FROM data.articles a, websearch_to_tsquery('english', in_query) AS q\n WHERE a.search_vector @@ q\n )\n\n ORDER BY score DESC\n LIMIT in_limit;\n$;\n```\n\n### Search Suggestions (Autocomplete)\n\n```sql\n-- Table for search terms\nCREATE TABLE data.search_terms (\n term text PRIMARY KEY,\n frequency integer NOT NULL DEFAULT 1,\n last_used timestamptz NOT NULL DEFAULT now()\n);\n\n-- Trigram index for prefix matching\nCREATE INDEX search_terms_trgm_idx ON data.search_terms USING gin(term gin_trgm_ops);\n\n-- Track searches\nCREATE PROCEDURE api.track_search(in_term text)\nLANGUAGE sql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n INSERT INTO data.search_terms (term, frequency, last_used)\n VALUES (lower(trim(in_term)), 1, now())\n ON CONFLICT (term) DO UPDATE\n SET frequency = search_terms.frequency + 1,\n last_used = now();\n$;\n\n-- Autocomplete function\nCREATE FUNCTION api.search_suggestions(in_prefix text, in_limit integer DEFAULT 10)\nRETURNS TABLE (term text, frequency integer)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT term, frequency\n FROM data.search_terms\n WHERE term LIKE lower(in_prefix) || '%'\n ORDER BY frequency DESC, last_used DESC\n LIMIT in_limit;\n$;\n```\n\n### Faceted Search\n\n```sql\n-- Get search results with category counts\nCREATE FUNCTION api.search_with_facets(in_query text)\nRETURNS TABLE (\n results jsonb,\n facets jsonb\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n WITH search_results AS (\n SELECT id, title, category, ts_rank(search_vector, q) AS rank\n FROM data.articles, websearch_to_tsquery('english', in_query) AS q\n WHERE search_vector @@ q\n ),\n result_data AS (\n SELECT jsonb_agg(\n jsonb_build_object('id', id, 'title', title, 'category', category)\n ORDER BY rank DESC\n ) AS results\n FROM (SELECT * FROM search_results LIMIT 20) sub\n ),\n facet_data AS (\n SELECT jsonb_object_agg(category, cnt) AS facets\n FROM (\n SELECT category, count(*) AS cnt\n FROM search_results\n GROUP BY category\n ORDER BY cnt DESC\n ) sub\n )\n SELECT result_data.results, facet_data.facets\n FROM result_data, facet_data;\n$;\n```\n\n## Performance Optimization\n\n### Index Maintenance\n\n```sql\n-- Check index size\nSELECT\n indexrelname,\n pg_size_pretty(pg_relation_size(indexrelid)) AS index_size\nFROM pg_stat_user_indexes\nWHERE indexrelname LIKE '%search%';\n\n-- Reindex if bloated\nREINDEX INDEX CONCURRENTLY articles_search_idx;\n```\n\n### Query Optimization\n\n```sql\n-- EXPLAIN ANALYZE your searches\nEXPLAIN (ANALYZE, BUFFERS)\nSELECT * FROM data.articles\nWHERE search_vector @@ to_tsquery('english', 'postgresql')\nORDER BY ts_rank(search_vector, to_tsquery('english', 'postgresql')) DESC\nLIMIT 20;\n\n-- Should show: Bitmap Index Scan on articles_search_idx\n```\n\n### Limit Result Set Early\n\n```sql\n-- BAD: Rank all matches then limit\nSELECT *, ts_rank(search_vector, q) AS rank\nFROM data.articles, to_tsquery('english', 'common') AS q\nWHERE search_vector @@ q\nORDER BY rank DESC\nLIMIT 20;\n\n-- BETTER: Use threshold to reduce ranking work\nSELECT *, ts_rank(search_vector, q) AS rank\nFROM data.articles, to_tsquery('english', 'common') AS q\nWHERE search_vector @@ q\n AND ts_rank(search_vector, q) > 0.01 -- Filter low-relevance early\nORDER BY rank DESC\nLIMIT 20;\n```\n\n### Avoid ts_headline on Large Result Sets\n\n```sql\n-- BAD: Generate snippets for all results\nSELECT id, title, ts_headline('english', body, q) AS snippet\nFROM data.articles, to_tsquery('english', 'postgresql') AS q\nWHERE search_vector @@ q;\n\n-- GOOD: Generate snippets only for displayed results\nWITH ranked AS (\n SELECT id, title, body, ts_rank(search_vector, q) AS rank\n FROM data.articles, to_tsquery('english', 'postgresql') AS q\n WHERE search_vector @@ q\n ORDER BY rank DESC\n LIMIT 20\n)\nSELECT\n id,\n title,\n ts_headline('english', body, to_tsquery('english', 'postgresql')) AS snippet\nFROM ranked;\n```\n\n### Statistics and Monitoring\n\n```sql\n-- Most common search terms (from pg_stat_statements)\nSELECT query, calls, mean_exec_time\nFROM pg_stat_statements\nWHERE query LIKE '%@@%tsquery%'\nORDER BY calls DESC\nLIMIT 20;\n\n-- tsvector statistics\nSELECT\n word,\n ndoc,\n nentry\nFROM ts_stat('SELECT search_vector FROM data.articles')\nORDER BY nentry DESC\nLIMIT 50;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":22341,"content_sha256":"d542f08e8c8a596c9f8db15e6cab537fe31dcd3b03dd4ee7adfb5740bef71558"},{"filename":"references/indexes-constraints.md","content":"# Indexes & Constraints Best Practices\n\n## Table of Contents\n1. [Index Fundamentals](#index-fundamentals)\n2. [Index Types](#index-types)\n3. [Index Design Strategies](#index-design-strategies)\n4. [Constraint Types](#constraint-types)\n5. [Foreign Key Design](#foreign-key-design)\n6. [PostgreSQL 18 Features](#postgresql-18-features)\n7. [Maintenance](#maintenance)\n\n## Index Fundamentals\n\n### Index Selection Decision Tree\n\n```mermaid\nflowchart TD\n START([Need to optimize a query?]) --> Q1{What operation?}\n \n Q1 -->|\"= equality\"| EQ{Data type?}\n Q1 -->|\"\u003c, >, BETWEEN\"| BTREE[\"B-tree Index\"]\n Q1 -->|\"ORDER BY\"| BTREE\n Q1 -->|\"LIKE 'prefix%'\"| BTREE\n Q1 -->|\"LIKE '%text%'\"| TRGM[\"pg_trgm + GIN\"]\n Q1 -->|\"Full-text search\"| GIN_FTS[\"GIN + tsvector\"]\n Q1 -->|\"Array contains @>\"| GIN_ARR[\"GIN\"]\n Q1 -->|\"JSONB contains\"| GIN_JSON[\"GIN\"]\n Q1 -->|\"Geometry/Range\"| GIST[\"GiST\"]\n \n EQ -->|\"Scalar (text, int, uuid)\"| BTREE\n EQ -->|\"Only equality, very large table\"| HASH[\"Hash Index\"]\n \n BTREE --> PARTIAL{Subset of rows?}\n PARTIAL -->|\"Yes\"| PARTIAL_IDX[\"Add WHERE clause\u003cbr/>(Partial Index)\"]\n PARTIAL -->|\"No\"| FULL_IDX[\"Full Index\"]\n \n style BTREE fill:#c8e6c9\n style GIN_FTS fill:#bbdefb\n style GIN_ARR fill:#bbdefb\n style GIN_JSON fill:#bbdefb\n style GIST fill:#fff3e0\n```\n\n### When to Create Indexes\n\n**Do index:**\n- Columns frequently used in WHERE clauses\n- Columns used in JOIN conditions\n- Columns used in ORDER BY\n- Foreign key columns (not automatic in PostgreSQL)\n- Columns with high selectivity (many distinct values)\n\n**Avoid over-indexing:**\n- Write-heavy tables (each index slows INSERT/UPDATE/DELETE)\n- Low-selectivity columns (boolean, status with few values)\n- Rarely queried columns\n- Small tables (sequential scan often faster)\n\n### Index Naming\n\n```sql\n-- Standard index: {table}_{columns}_idx (Trivadis v4.4)\nCREATE INDEX orders_customer_id_idx ON data.orders(customer_id);\n\n-- Multi-column: {table}_{col1}_{col2}_idx\nCREATE INDEX orders_status_created_idx ON data.orders(status, created_at);\n\n-- Unique index: {table}_{columns}_key\nCREATE UNIQUE INDEX users_email_key ON data.users(lower(email));\n\n-- Partial index: {table}_{column}_{condition_hint}_idx\nCREATE INDEX orders_pending_idx ON data.orders(created_at) WHERE status = 'pending';\n```\n\n## Index Types\n\n### Index Type Comparison\n\n```mermaid\ngraph TB\n subgraph BTREE[\"B-tree (Default)\"]\n B1[\"✓ =, \u003c, >, \u003c=, >=, BETWEEN\"]\n B2[\"✓ ORDER BY\"]\n B3[\"✓ LIKE 'prefix%'\"]\n B4[\"✗ LIKE '%suffix'\"]\n end\n \n subgraph HASH[\"Hash\"]\n H1[\"✓ = only\"]\n H2[\"✓ Faster for equality\"]\n H3[\"✗ No range queries\"]\n H4[\"✗ No sorting\"]\n end\n \n subgraph GIN[\"GIN\"]\n G1[\"✓ Array contains\"]\n G2[\"✓ JSONB contains\"]\n G3[\"✓ Full-text search\"]\n G4[\"✗ Slower writes\"]\n end\n \n subgraph GIST[\"GiST\"]\n GI1[\"✓ Geometry\"]\n GI2[\"✓ Range types\"]\n GI3[\"✓ Full-text (slower)\"]\n GI4[\"✓ Exclusion constraints\"]\n end\n \n subgraph BRIN[\"BRIN\"]\n BR1[\"✓ Very large tables\"]\n BR2[\"✓ Naturally ordered data\"]\n BR3[\"✓ Tiny size\"]\n BR4[\"✗ Less precise\"]\n end\n \n style BTREE fill:#c8e6c9\n style GIN fill:#bbdefb\n style GIST fill:#fff3e0\n style BRIN fill:#f3e5f5\n```\n\n### B-tree (Default)\n\nBest for: equality, range queries, sorting\n\n```sql\n-- Equality\nCREATE INDEX users_email_idx ON data.users(email);\nSELECT * FROM data.users WHERE email = '[email protected]';\n\n-- Range\nCREATE INDEX orders_created_idx ON data.orders(created_at);\nSELECT * FROM data.orders WHERE created_at >= '2024-01-01';\n\n-- Sorting\nCREATE INDEX orders_created_desc_idx ON data.orders(created_at DESC);\nSELECT * FROM data.orders ORDER BY created_at DESC LIMIT 10;\n\n-- Multi-column (column order matters!)\nCREATE INDEX orders_customer_date_idx ON data.orders(customer_id, created_at DESC);\n-- Efficient for: WHERE customer_id = X ORDER BY created_at DESC\n-- Also works for: WHERE customer_id = X (uses first column)\n-- Skip scan (PG18): WHERE created_at > X (can use second column)\n```\n\n### Hash\n\nBest for: equality only, faster than B-tree for simple lookups\n\n```sql\nCREATE INDEX users_email_hash_idx ON data.users USING hash(email);\n\n-- Only useful for equality\nSELECT * FROM data.users WHERE email = '[email protected]'; -- Uses hash\nSELECT * FROM data.users WHERE email LIKE 'user%'; -- Cannot use hash\n```\n\n### GIN (Generalized Inverted Index)\n\nBest for: arrays, JSONB, full-text search\n\n```sql\n-- Array containment\nCREATE INDEX products_tags_idx ON data.products USING gin(tags);\nSELECT * FROM data.products WHERE tags @> ARRAY['sale'];\n\n-- JSONB queries\nCREATE INDEX products_data_idx ON data.products USING gin(data);\nSELECT * FROM data.products WHERE data @> '{\"category\": \"electronics\"}';\n\n-- Full-text search\nCREATE INDEX articles_search_idx ON data.articles USING gin(to_tsvector('english', title || ' ' || body));\nSELECT * FROM data.articles WHERE to_tsvector('english', title || ' ' || body) @@ to_tsquery('postgresql & index');\n```\n\n### GiST (Generalized Search Tree)\n\nBest for: geometric data, range types, full-text search\n\n```sql\n-- Range overlaps (for temporal data)\nCREATE INDEX reservations_during_idx ON data.reservations USING gist(during);\nSELECT * FROM data.reservations WHERE during && tstzrange('2024-03-01', '2024-03-05');\n\n-- Geometric (PostGIS)\nCREATE INDEX locations_geom_idx ON data.locations USING gist(geom);\nSELECT * FROM data.locations WHERE ST_DWithin(geom, ST_MakePoint(-122.4, 37.8), 1000);\n```\n\n### BRIN (Block Range Index)\n\nBest for: large tables with naturally ordered data (time-series)\n\n```sql\n-- Very compact index for time-series data\nCREATE INDEX events_created_brin_idx ON data.events USING brin(created_at);\n\n-- Best when data is physically ordered by the indexed column\n-- Much smaller than B-tree but less precise\n```\n\n## Index Design Strategies\n\n### Covering Indexes with INCLUDE (PostgreSQL 11+)\n\nInclude all columns needed by query to avoid table access (index-only scans):\n\n```sql\n-- Basic covering index\n-- Index key: customer_id (used for filtering/sorting)\n-- Included: status, total, created_at (returned but not used for lookup)\nCREATE INDEX orders_customer_covering_idx\n ON data.orders(customer_id)\n INCLUDE (status, total, created_at);\n\n-- Query satisfied entirely from index (no heap access)\nSELECT customer_id, status, total\nFROM data.orders\nWHERE customer_id = 'uuid-here';\n```\n\n### INCLUDE vs Multi-Column Index\n\n```sql\n-- Multi-column index: All columns in B-tree structure\nCREATE INDEX orders_customer_status_idx ON data.orders(customer_id, status);\n-- Can be used for: WHERE customer_id = X\n-- WHERE customer_id = X AND status = Y\n-- ORDER BY customer_id, status\n\n-- INCLUDE index: Only key columns in B-tree, included columns stored separately\nCREATE INDEX orders_customer_include_idx ON data.orders(customer_id) INCLUDE (status);\n-- Can be used for: WHERE customer_id = X (returns status without heap access)\n-- Cannot be used for: WHERE status = Y (status not in B-tree)\n```\n\n### When to Use INCLUDE\n\n```sql\n-- 1. High cardinality key + low cardinality included columns\nCREATE INDEX orders_customer_idx ON data.orders(customer_id)\n INCLUDE (status, total); -- status has few distinct values\n\n-- 2. Avoid adding large columns to B-tree\nCREATE INDEX products_sku_idx ON data.products(sku)\n INCLUDE (name, description); -- Don't want text in B-tree\n\n-- 3. Unique constraint that returns additional columns\nCREATE UNIQUE INDEX users_email_key ON data.users(email)\n INCLUDE (id, name); -- Can return id, name in index-only scan\n\n-- 4. Foreign key lookups returning related data\nCREATE INDEX order_items_order_idx ON data.order_items(order_id)\n INCLUDE (product_id, quantity, price);\n```\n\n### Verify Index-Only Scans\n\n```sql\n-- EXPLAIN should show \"Index Only Scan\"\nEXPLAIN (ANALYZE)\nSELECT customer_id, status, total\nFROM data.orders\nWHERE customer_id = 'uuid-here';\n\n-- If showing \"Index Scan\" instead of \"Index Only Scan\":\n-- 1. Ensure all SELECTed columns are in INCLUDE\n-- 2. Run VACUUM to update visibility map\nVACUUM data.orders;\n```\n\n### Partial Indexes\n\nIndex only rows matching a condition:\n\n```sql\n-- Only index active users\nCREATE INDEX users_email_active_idx\n ON data.users(email)\n WHERE is_active = true;\n\n-- Only index pending orders\nCREATE INDEX orders_pending_idx\n ON data.orders(customer_id, created_at)\n WHERE status = 'pending';\n\n-- Index for soft-deleted records lookup\nCREATE INDEX customers_deleted_idx\n ON data.customers(deleted_at)\n WHERE deleted_at IS NOT NULL;\n```\n\n### Expression Indexes\n\nIndex computed values:\n\n```sql\n-- Case-insensitive email lookup\nCREATE INDEX users_email_lower_idx ON data.users(lower(email));\nSELECT * FROM data.users WHERE lower(email) = lower('[email protected]');\n\n-- Year from timestamp\nCREATE INDEX orders_year_idx ON data.orders((extract(year from created_at)));\nSELECT * FROM data.orders WHERE extract(year from created_at) = 2024;\n\n-- JSONB expression\nCREATE INDEX products_category_idx ON data.products((data->>'category'));\nSELECT * FROM data.products WHERE data->>'category' = 'electronics';\n```\n\n### Multi-Column Index Column Order\n\nThe order of columns matters for query optimization:\n\n```sql\n-- Index: (status, customer_id, created_at)\n\n-- ✅ Uses full index\nWHERE status = 'pending' AND customer_id = X AND created_at > Y\n\n-- ✅ Uses first two columns \nWHERE status = 'pending' AND customer_id = X\n\n-- ✅ Uses first column\nWHERE status = 'pending'\n\n-- ⚠️ PostgreSQL 18 skip scan can help\nWHERE customer_id = X -- Skips through status values\n\n-- ❌ Cannot use index efficiently (before PG18)\nWHERE created_at > Y -- Without status filter\n```\n\n### Concurrent Index Creation\n\nCreate indexes without blocking writes:\n\n```sql\n-- Non-blocking index creation (takes longer, allows writes)\nCREATE INDEX CONCURRENTLY orders_customer_idx\n ON data.orders(customer_id);\n\n-- Note: CONCURRENTLY cannot be used in transactions\n-- If creation fails, index may be left invalid - check and retry:\nSELECT indexrelid::regclass, indisvalid \nFROM pg_index \nWHERE NOT indisvalid;\n\n-- Rebuild invalid index\nREINDEX INDEX CONCURRENTLY orders_customer_idx;\n```\n\n## Constraint Types\n\n### Primary Key\n\n```sql\n-- Inline definition\nCREATE TABLE data.orders (\n id uuid PRIMARY KEY DEFAULT uuidv7()\n);\n\n-- Named constraint\nCREATE TABLE data.orders (\n id uuid DEFAULT uuidv7(),\n CONSTRAINT orders_pkey PRIMARY KEY (id)\n);\n\n-- Composite primary key\nCREATE TABLE data.order_items (\n order_id uuid,\n item_number integer,\n CONSTRAINT order_items_pkey PRIMARY KEY (order_id, item_number)\n);\n```\n\n### Unique Constraints\n\n```sql\n-- Simple unique\nALTER TABLE data.users \n ADD CONSTRAINT users_email_key UNIQUE (email);\n\n-- Case-insensitive unique (use index instead)\nCREATE UNIQUE INDEX users_email_lower_key ON data.users(lower(email));\n\n-- Partial unique (unique among active only)\nCREATE UNIQUE INDEX users_username_active_key \n ON data.users(username) \n WHERE is_active = true;\n\n-- Unique with nulls distinct (default in PG15+)\nALTER TABLE data.products \n ADD CONSTRAINT products_sku_key UNIQUE NULLS DISTINCT (sku);\n```\n\n### Check Constraints\n\n```sql\n-- Value range\nALTER TABLE data.orders \n ADD CONSTRAINT orders_total_positive CHECK (total >= 0);\n\n-- Enumeration\nALTER TABLE data.orders \n ADD CONSTRAINT orders_status_check \n CHECK (status IN ('draft', 'pending', 'confirmed', 'shipped', 'delivered', 'cancelled'));\n\n-- Pattern match\nALTER TABLE data.users \n ADD CONSTRAINT users_email_format_check \n CHECK (email ~* '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z]{2,}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

);\n\n-- Multi-column check\nALTER TABLE data.reservations \n ADD CONSTRAINT reservations_dates_check \n CHECK (check_out > check_in);\n\n-- NOT VALID: Add constraint without scanning existing rows\nALTER TABLE data.large_table \n ADD CONSTRAINT large_table_value_check CHECK (value > 0) NOT VALID;\n\n-- Validate later (can run concurrently)\nALTER TABLE data.large_table VALIDATE CONSTRAINT large_table_value_check;\n```\n\n### Not Null Constraints\n\n```sql\n-- Standard NOT NULL\nALTER TABLE data.users ALTER COLUMN email SET NOT NULL;\n\n-- NOT NULL with NOT VALID (PG18+): No immediate table scan\nALTER TABLE data.large_table \n ADD CONSTRAINT large_table_name_not_null \n CHECK (name IS NOT NULL) NOT VALID;\n\n-- Validate later\nALTER TABLE data.large_table VALIDATE CONSTRAINT large_table_name_not_null;\n```\n\n### Exclusion Constraints\n\nPrevent overlapping values:\n\n```sql\n-- Prevent overlapping reservations\nCREATE TABLE data.reservations (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n room_id uuid NOT NULL,\n during tstzrange NOT NULL,\n \n CONSTRAINT reservations_no_overlap \n EXCLUDE USING gist (room_id WITH =, during WITH &&)\n);\n\n-- Prevent overlapping employee assignments\nCREATE TABLE data.assignments (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n employee_id uuid NOT NULL,\n project_id uuid NOT NULL,\n during daterange NOT NULL,\n \n CONSTRAINT assignments_no_overlap\n EXCLUDE USING gist (employee_id WITH =, during WITH &&)\n);\n```\n\n## Foreign Key Design\n\n### Basic Foreign Keys\n\n```sql\nCREATE TABLE data.orders (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n customer_id uuid NOT NULL,\n \n CONSTRAINT orders_customer_fkey \n FOREIGN KEY (customer_id) REFERENCES data.customers(id)\n);\n\n-- Always index foreign key columns\nCREATE INDEX orders_customer_id_idx ON data.orders(customer_id);\n```\n\n### Referential Actions\n\n```sql\n-- ON DELETE options:\n-- - RESTRICT: Prevent deletion if referenced (default)\n-- - CASCADE: Delete referencing rows\n-- - SET NULL: Set FK to NULL\n-- - SET DEFAULT: Set FK to default value\n-- - NO ACTION: Check at end of transaction (allows deferred)\n\n-- ON UPDATE options (same as above)\n\n-- Example: Cascade delete order items when order deleted\nALTER TABLE data.order_items \n ADD CONSTRAINT order_items_order_fkey \n FOREIGN KEY (order_id) REFERENCES data.orders(id) \n ON DELETE CASCADE;\n\n-- Example: Set null when customer deleted (keep orphaned orders)\nALTER TABLE data.orders \n ADD CONSTRAINT orders_customer_fkey \n FOREIGN KEY (customer_id) REFERENCES data.customers(id) \n ON DELETE SET NULL;\n\n-- Example: Restrict deletion\nALTER TABLE data.accounts \n ADD CONSTRAINT accounts_user_fkey \n FOREIGN KEY (user_id) REFERENCES data.users(id) \n ON DELETE RESTRICT;\n```\n\n### Self-Referential Foreign Keys\n\n```sql\n-- Tree structure (parent-child)\nCREATE TABLE data.categories (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n parent_id uuid,\n \n CONSTRAINT categories_parent_fkey \n FOREIGN KEY (parent_id) REFERENCES data.categories(id) \n ON DELETE CASCADE\n);\n\nCREATE INDEX categories_parent_id_idx ON data.categories(parent_id);\n```\n\n### Deferrable Constraints\n\n```sql\n-- Allow circular references within transaction\nCREATE TABLE data.employees (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n manager_id uuid,\n \n CONSTRAINT employees_manager_fkey \n FOREIGN KEY (manager_id) REFERENCES data.employees(id)\n DEFERRABLE INITIALLY DEFERRED\n);\n\n-- Insert circular reference in transaction\nBEGIN;\nINSERT INTO data.employees (id, manager_id) VALUES ('a', 'b');\nINSERT INTO data.employees (id, manager_id) VALUES ('b', 'a');\nCOMMIT; -- Constraint checked here\n```\n\n## PostgreSQL 18 Features\n\n### Temporal Constraints\n\nPrimary keys, unique constraints, and foreign keys with temporal ranges:\n\n```sql\n-- Temporal primary key: no overlapping time ranges for same entity\nCREATE TABLE data.employee_positions (\n employee_id uuid NOT NULL,\n position_id uuid NOT NULL,\n valid_during daterange NOT NULL,\n \n -- Temporal primary key (PG18)\n CONSTRAINT employee_positions_pk \n PRIMARY KEY (employee_id, valid_during WITHOUT OVERLAPS)\n);\n\n-- Temporal unique constraint\nCREATE TABLE data.room_rates (\n room_type text NOT NULL,\n rate numeric(10,2) NOT NULL,\n valid_during daterange NOT NULL,\n \n CONSTRAINT room_rates_unique \n UNIQUE (room_type, valid_during WITHOUT OVERLAPS)\n);\n\n-- Temporal foreign key\nCREATE TABLE data.bookings (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n room_type text NOT NULL,\n booking_period daterange NOT NULL,\n \n CONSTRAINT bookings_rate_fkey \n FOREIGN KEY (room_type, PERIOD booking_period) \n REFERENCES data.room_rates (room_type, PERIOD valid_during)\n);\n```\n\n### NOT NULL with NOT VALID\n\nAdd NOT NULL constraints without blocking table:\n\n```sql\n-- Add constraint without scanning table\nALTER TABLE data.large_table \n ADD CONSTRAINT large_table_name_nn CHECK (name IS NOT NULL) NOT VALID;\n\n-- Validate in background (allows concurrent access)\nALTER TABLE data.large_table VALIDATE CONSTRAINT large_table_name_nn;\n```\n\n### Skip Scan for Multi-Column Indexes\n\nPostgreSQL 18 can skip leading columns in multi-column B-tree indexes:\n\n```sql\n-- Index on (tenant_id, user_id, created_at)\nCREATE INDEX events_tenant_user_created_idx\n ON data.events(tenant_id, user_id, created_at);\n\n-- Before PG18: Required tenant_id filter to use index\n-- PG18: Can skip tenant_id and use user_id filter\nSELECT * FROM data.events WHERE user_id = 'uuid' ORDER BY created_at;\n```\n\n## Maintenance\n\n### Monitor Index Usage\n\n```sql\n-- Find unused indexes\nSELECT \n schemaname,\n relname AS table_name,\n indexrelname AS index_name,\n idx_scan AS times_used,\n pg_size_pretty(pg_relation_size(indexrelid)) AS index_size\nFROM pg_stat_user_indexes\nWHERE idx_scan = 0\n AND schemaname NOT IN ('pg_catalog', 'pg_toast')\nORDER BY pg_relation_size(indexrelid) DESC;\n\n-- Index hit ratio (should be > 99%)\nSELECT \n relname,\n round(100.0 * idx_blks_hit / nullif(idx_blks_hit + idx_blks_read, 0), 2) AS hit_ratio\nFROM pg_statio_user_indexes\nWHERE idx_blks_hit + idx_blks_read > 0\nORDER BY hit_ratio;\n```\n\n### Reindexing\n\n```sql\n-- Rebuild single index (blocking)\nREINDEX INDEX data.orders_customer_id_idx;\n\n-- Rebuild all indexes on table (blocking)\nREINDEX TABLE data.orders;\n\n-- Rebuild without blocking (PG12+)\nREINDEX INDEX CONCURRENTLY data.orders_customer_id_idx;\n\n-- Rebuild all indexes in schema concurrently\nREINDEX SCHEMA CONCURRENTLY app;\n```\n\n### Index Bloat Detection\n\n```sql\n-- Check for bloated indexes\nSELECT\n schemaname || '.' || relname AS table_name,\n indexrelname AS index_name,\n pg_size_pretty(pg_relation_size(indexrelid)) AS index_size,\n pg_size_pretty(pg_relation_size(relid)) AS table_size,\n round(100.0 * pg_relation_size(indexrelid) / nullif(pg_relation_size(relid), 0), 1) AS index_ratio\nFROM pg_stat_user_indexes\nWHERE schemaname = 'app'\nORDER BY pg_relation_size(indexrelid) DESC\nLIMIT 20;\n```\n\n### Statistics Updates\n\n```sql\n-- Update statistics for better query planning\nANALYZE data.orders;\n\n-- Update all tables in schema\nDO $\nDECLARE\n r RECORD;\nBEGIN\n FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = 'app'\n LOOP\n EXECUTE 'ANALYZE data.' || quote_ident(r.tablename);\n END LOOP;\nEND;\n$;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":19192,"content_sha256":"d5047810535204ceecd4e1d845aa853c7f1df82076f1a18be4d9b1ac9141018f"},{"filename":"references/jsonb-patterns.md","content":"# JSON/JSONB Patterns\n\nThis document covers when to use JSONB, indexing strategies, common query patterns, and validation techniques for semi-structured data in PostgreSQL.\n\n## Table of Contents\n\n1. [When to Use JSONB](#when-to-use-jsonb)\n2. [JSONB vs Normalized Tables](#jsonb-vs-normalized-tables)\n3. [Schema Design with JSONB](#schema-design-with-jsonb)\n4. [Indexing Strategies](#indexing-strategies)\n5. [Query Patterns](#query-patterns)\n6. [Validation Patterns](#validation-patterns)\n7. [Update Patterns](#update-patterns)\n8. [Performance Optimization](#performance-optimization)\n\n## When to Use JSONB\n\n### Good Use Cases\n\n| Use Case | Example |\n|----------|---------|\n| **Flexible attributes** | Product metadata, user preferences |\n| **Event data** | Audit logs, analytics events |\n| **External API responses** | Cached webhook payloads |\n| **Document storage** | CMS content, form submissions |\n| **Schema-less MVP** | Rapid prototyping |\n| **Sparse data** | Optional fields that vary widely |\n\n### Avoid JSONB When\n\n| Situation | Better Alternative |\n|-----------|-------------------|\n| Fixed, known schema | Regular columns |\n| Frequent joins on values | Normalized tables with FK |\n| Aggregate queries | Numeric columns |\n| Unique constraints | Regular columns + indexes |\n| Foreign key relationships | Proper FK columns |\n\n### Decision Flowchart\n\n```mermaid\nflowchart TD\n START([Need to store data?]) --> Q1{Schema known\u003cbr/>and fixed?}\n \n Q1 -->|Yes| Q2{Queried frequently\u003cbr/>in WHERE/JOIN?}\n Q1 -->|No| JSONB[\"Use JSONB\"]\n \n Q2 -->|Yes| COLUMNS[\"Use regular columns\"]\n Q2 -->|No| Q3{Data sparse?\u003cbr/>Many NULLs?}\n \n Q3 -->|Yes| JSONB\n Q3 -->|No| COLUMNS\n \n style JSONB fill:#bbdefb\n style COLUMNS fill:#c8e6c9\n```\n\n## JSONB vs Normalized Tables\n\n### Normalized Approach (Traditional)\n\n```sql\n-- Separate tables for product attributes\nCREATE TABLE data.products (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n price numeric(10,2) NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE TABLE data.product_attributes (\n product_id uuid REFERENCES data.products(id) ON DELETE CASCADE,\n key text NOT NULL,\n value text NOT NULL,\n PRIMARY KEY (product_id, key)\n);\n\n-- Query: Find products with color = 'red'\nSELECT p.* \nFROM data.products p\nJOIN data.product_attributes pa ON pa.product_id = p.id\nWHERE pa.key = 'color' AND pa.value = 'red';\n```\n\n### JSONB Approach\n\n```sql\n-- Single table with JSONB attributes\nCREATE TABLE data.products (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n price numeric(10,2) NOT NULL,\n attributes jsonb NOT NULL DEFAULT '{}',\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Query: Find products with color = 'red'\nSELECT * FROM data.products\nWHERE attributes->>'color' = 'red';\n\n-- Or using containment\nSELECT * FROM data.products\nWHERE attributes @> '{\"color\": \"red\"}';\n```\n\n### Comparison\n\n| Aspect | Normalized | JSONB |\n|--------|------------|-------|\n| **Schema flexibility** | Rigid | Flexible |\n| **Query performance** | Excellent with indexes | Good with GIN indexes |\n| **Storage efficiency** | Better for sparse data | Better for dense data |\n| **Referential integrity** | FK constraints | Manual validation |\n| **Aggregations** | Native SQL | Requires extraction |\n| **Tooling support** | Excellent | Good |\n\n## Schema Design with JSONB\n\n### Hybrid Approach (Recommended)\n\n```sql\n-- Core fields as columns, flexible data as JSONB\nCREATE TABLE data.orders (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n customer_id uuid NOT NULL REFERENCES data.customers(id),\n status text NOT NULL DEFAULT 'pending',\n total numeric(10,2) NOT NULL,\n \n -- Structured JSONB for known flexible data\n shipping_address jsonb NOT NULL,\n \n -- Unstructured JSONB for truly flexible data\n metadata jsonb NOT NULL DEFAULT '{}',\n \n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- shipping_address structure:\n-- {\n-- \"street\": \"123 Main St\",\n-- \"city\": \"New York\",\n-- \"state\": \"NY\",\n-- \"zip\": \"10001\",\n-- \"country\": \"US\"\n-- }\n\n-- metadata can contain anything:\n-- {\n-- \"source\": \"mobile_app\",\n-- \"campaign_id\": \"summer2024\",\n-- \"notes\": \"Gift wrap requested\"\n-- }\n```\n\n### Event/Audit Log Pattern\n\n```sql\nCREATE TABLE data.events (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n event_type text NOT NULL,\n entity_type text NOT NULL,\n entity_id uuid NOT NULL,\n actor_id uuid,\n payload jsonb NOT NULL,\n occurred_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Index for common queries\nCREATE INDEX events_entity_idx ON data.events(entity_type, entity_id);\nCREATE INDEX events_type_idx ON data.events(event_type);\nCREATE INDEX events_occurred_idx ON data.events(occurred_at);\n\n-- GIN index for payload searches\nCREATE INDEX events_payload_idx ON data.events USING gin(payload);\n\n-- Example events:\n-- { \"event_type\": \"order.created\", \"entity_type\": \"order\", \"payload\": {\"total\": 99.99} }\n-- { \"event_type\": \"order.shipped\", \"entity_type\": \"order\", \"payload\": {\"carrier\": \"UPS\", \"tracking\": \"1Z...\"} }\n```\n\n### User Preferences Pattern\n\n```sql\nCREATE TABLE data.user_preferences (\n user_id uuid PRIMARY KEY REFERENCES data.users(id),\n preferences jsonb NOT NULL DEFAULT '{\n \"notifications\": {\n \"email\": true,\n \"push\": true,\n \"sms\": false\n },\n \"theme\": \"system\",\n \"language\": \"en\",\n \"timezone\": \"UTC\"\n }',\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Merge new preferences (partial update)\nUPDATE data.user_preferences\nSET preferences = preferences || '{\"theme\": \"dark\"}'::jsonb,\n updated_at = now()\nWHERE user_id = $1;\n```\n\n## Indexing Strategies\n\n### GIN Index (Default for JSONB)\n\n```sql\n-- Index entire JSONB document\nCREATE INDEX products_attributes ON data.products USING gin(attributes);\n\n-- Supports operators: @>, ?, ?&, ?|, @?\n-- @> containment\n-- ? key exists\n-- ?& all keys exist\n-- ?| any key exists\n-- @? JSON path exists\n```\n\n### GIN with jsonb_path_ops (Faster Containment)\n\n```sql\n-- More efficient for @> containment queries only\nCREATE INDEX products_attrs_path \n ON data.products USING gin(attributes jsonb_path_ops);\n\n-- 2-3x smaller index, faster @> queries\n-- Does NOT support ?, ?&, ?| operators\n\n-- Use when you primarily search with @>\nSELECT * FROM data.products \nWHERE attributes @> '{\"color\": \"red\", \"size\": \"large\"}';\n```\n\n### B-tree on Extracted Values\n\n```sql\n-- Index specific JSON path as regular B-tree\nCREATE INDEX products_color \n ON data.products ((attributes->>'color'));\n\n-- Supports =, \u003c, >, LIKE, etc.\nSELECT * FROM data.products \nWHERE attributes->>'color' = 'red';\n\n-- Index with type cast for proper comparison\nCREATE INDEX products_weight \n ON data.products (((attributes->>'weight')::numeric));\n\nSELECT * FROM data.products \nWHERE (attributes->>'weight')::numeric > 10;\n```\n\n### Expression Index for Nested Paths\n\n```sql\n-- Index deeply nested value\nCREATE INDEX orders_shipping_city \n ON data.orders ((shipping_address->>'city'));\n\n-- Or using path extraction\nCREATE INDEX orders_shipping_zip \n ON data.orders ((shipping_address #>> '{zip}'));\n\n-- Query uses the index\nSELECT * FROM data.orders \nWHERE shipping_address->>'city' = 'New York';\n```\n\n### Partial Index on JSONB Condition\n\n```sql\n-- Index only products with 'featured' flag\nCREATE INDEX products_featured \n ON data.products ((attributes->>'category'))\n WHERE attributes @> '{\"featured\": true}';\n\n-- Query benefits from smaller index\nSELECT * FROM data.products \nWHERE attributes @> '{\"featured\": true}'\n AND attributes->>'category' = 'electronics';\n```\n\n## Query Patterns\n\n### Basic Extraction\n\n```sql\n-- Get single value as text\nSELECT attributes->>'color' AS color FROM data.products;\n\n-- Get single value preserving JSON type\nSELECT attributes->'price' AS price FROM data.products; -- Returns JSON\n\n-- Get nested value\nSELECT shipping_address->'coordinates'->>'lat' AS latitude FROM data.orders;\n\n-- Path extraction (alternative syntax)\nSELECT attributes #>> '{dimensions,width}' AS width FROM data.products;\n```\n\n### Containment Queries\n\n```sql\n-- Contains key-value pair\nSELECT * FROM data.products \nWHERE attributes @> '{\"color\": \"red\"}';\n\n-- Contains multiple conditions\nSELECT * FROM data.products \nWHERE attributes @> '{\"color\": \"red\", \"size\": \"large\"}';\n\n-- Contains nested structure\nSELECT * FROM data.orders \nWHERE shipping_address @> '{\"country\": \"US\", \"state\": \"NY\"}';\n```\n\n### Key Existence\n\n```sql\n-- Has specific key\nSELECT * FROM data.products \nWHERE attributes ? 'color';\n\n-- Has all keys\nSELECT * FROM data.products \nWHERE attributes ?& array['color', 'size'];\n\n-- Has any key\nSELECT * FROM data.products \nWHERE attributes ?| array['discount', 'sale_price'];\n```\n\n### JSON Path Queries (PostgreSQL 12+)\n\n```sql\n-- Check if path exists with condition\nSELECT * FROM data.products \nWHERE attributes @? '$.tags[*] ? (@ == \"sale\")';\n\n-- Extract matching values\nSELECT jsonb_path_query_array(attributes, '$.tags[*] ? (@ like_regex \"^s\")')\nFROM data.products;\n\n-- Filter with path expression\nSELECT * FROM data.products \nWHERE jsonb_path_exists(attributes, '$.price ? (@ > 100)');\n```\n\n### Aggregation with JSONB\n\n```sql\n-- Count by JSON field\nSELECT \n attributes->>'category' AS category,\n COUNT(*) AS product_count,\n AVG((attributes->>'price')::numeric) AS avg_price\nFROM data.products\nGROUP BY attributes->>'category';\n\n-- Aggregate JSON values into array\nSELECT \n customer_id,\n jsonb_agg(jsonb_build_object(\n 'order_id', id,\n 'total', total,\n 'status', status\n )) AS orders\nFROM data.orders\nGROUP BY customer_id;\n```\n\n### Expanding JSONB Arrays\n\n```sql\n-- Sample data: attributes = '{\"tags\": [\"red\", \"sale\", \"new\"]}'\n\n-- Expand array to rows\nSELECT p.id, p.name, tag.value AS tag\nFROM data.products p,\n jsonb_array_elements_text(p.attributes->'tags') AS tag(value);\n\n-- Find products with specific tag\nSELECT * FROM data.products\nWHERE attributes->'tags' ? 'sale';\n\n-- Or using containment\nSELECT * FROM data.products\nWHERE attributes @> '{\"tags\": [\"sale\"]}';\n```\n\n### Building JSONB\n\n```sql\n-- Build object\nSELECT jsonb_build_object(\n 'id', id,\n 'name', name,\n 'attributes', attributes\n) AS product_json\nFROM data.products;\n\n-- Build array\nSELECT jsonb_agg(name) AS product_names\nFROM data.products\nWHERE attributes->>'category' = 'electronics';\n\n-- Combine objects\nSELECT \n to_jsonb(p.*) || jsonb_build_object('category_name', c.name) AS enriched\nFROM data.products p\nJOIN data.categories c ON c.id = (p.attributes->>'category_id')::uuid;\n```\n\n## Validation Patterns\n\n### CHECK Constraint Validation\n\n```sql\n-- Require specific keys\nALTER TABLE data.orders ADD CONSTRAINT orders_shipping_required\n CHECK (\n shipping_address ? 'street' AND\n shipping_address ? 'city' AND\n shipping_address ? 'country'\n );\n\n-- Validate value format\nALTER TABLE data.orders ADD CONSTRAINT orders_shipping_country_format\n CHECK (\n length(shipping_address->>'country') = 2 -- ISO country code\n );\n\n-- Validate nested structure\nALTER TABLE data.products ADD CONSTRAINT products_dimensions_valid\n CHECK (\n attributes->'dimensions' IS NULL OR (\n attributes->'dimensions' ? 'width' AND\n attributes->'dimensions' ? 'height' AND\n (attributes->'dimensions'->>'width')::numeric > 0 AND\n (attributes->'dimensions'->>'height')::numeric > 0\n )\n );\n```\n\n### Validation Function\n\n```sql\nCREATE OR REPLACE FUNCTION private.validate_shipping_address(in_address jsonb)\nRETURNS boolean\nLANGUAGE plpgsql\nIMMUTABLE\nAS $\nBEGIN\n -- Required fields\n IF NOT (in_address ? 'street' AND in_address ? 'city' AND in_address ? 'country') THEN\n RETURN false;\n END IF;\n \n -- Country must be 2-letter code\n IF length(in_address->>'country') != 2 THEN\n RETURN false;\n END IF;\n \n -- ZIP required for US\n IF in_address->>'country' = 'US' AND NOT in_address ? 'zip' THEN\n RETURN false;\n END IF;\n \n RETURN true;\nEND;\n$;\n\n-- Use in constraint\nALTER TABLE data.orders ADD CONSTRAINT orders_shipping_valid\n CHECK (private.validate_shipping_address(shipping_address));\n```\n\n### JSON Schema Validation (Extension)\n\n```sql\n-- Install pg_jsonschema extension (if available)\nCREATE EXTENSION IF NOT EXISTS pg_jsonschema;\n\n-- Define schema\nCREATE TABLE data.form_schemas (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL UNIQUE,\n schema jsonb NOT NULL\n);\n\nINSERT INTO data.form_schemas (name, schema) VALUES (\n 'contact_form',\n '{\n \"type\": \"object\",\n \"required\": [\"name\", \"email\", \"message\"],\n \"properties\": {\n \"name\": {\"type\": \"string\", \"minLength\": 1},\n \"email\": {\"type\": \"string\", \"format\": \"email\"},\n \"message\": {\"type\": \"string\", \"minLength\": 10}\n }\n }'\n);\n\n-- Validate on insert\nCREATE OR REPLACE FUNCTION private.validate_form_submission()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_schema jsonb;\nBEGIN\n SELECT schema INTO l_schema\n FROM data.form_schemas\n WHERE name = NEW.form_type;\n \n IF NOT jsonb_matches_schema(l_schema, NEW.data) THEN\n RAISE EXCEPTION 'Form data does not match schema for %', NEW.form_type;\n END IF;\n \n RETURN NEW;\nEND;\n$;\n```\n\n## Update Patterns\n\n### Replace Entire Value\n\n```sql\n-- Replace entire JSONB column\nUPDATE data.products\nSET attributes = '{\"color\": \"blue\", \"size\": \"medium\"}'::jsonb\nWHERE id = $1;\n```\n\n### Merge/Patch (Top-Level)\n\n```sql\n-- Merge with || (top-level keys only)\nUPDATE data.products\nSET attributes = attributes || '{\"color\": \"blue\"}'::jsonb\nWHERE id = $1;\n\n-- Result: existing keys preserved, color updated/added\n```\n\n### Set Nested Path\n\n```sql\n-- Set deeply nested value with jsonb_set\nUPDATE data.orders\nSET shipping_address = jsonb_set(\n shipping_address,\n '{coordinates,lat}',\n '40.7128'::jsonb\n)\nWHERE id = $1;\n\n-- Create missing path with create_if_missing = true\nUPDATE data.products\nSET attributes = jsonb_set(\n attributes,\n '{dimensions,depth}',\n '10'::jsonb,\n true -- create_if_missing\n)\nWHERE id = $1;\n```\n\n### Remove Key\n\n```sql\n-- Remove top-level key\nUPDATE data.products\nSET attributes = attributes - 'deprecated_field'\nWHERE id = $1;\n\n-- Remove nested key\nUPDATE data.products\nSET attributes = attributes #- '{dimensions,depth}'\nWHERE id = $1;\n\n-- Remove multiple keys\nUPDATE data.products\nSET attributes = attributes - 'key1' - 'key2'\nWHERE id = $1;\n```\n\n### Array Operations\n\n```sql\n-- Append to array\nUPDATE data.products\nSET attributes = jsonb_set(\n attributes,\n '{tags}',\n (attributes->'tags') || '\"new_tag\"'::jsonb\n)\nWHERE id = $1;\n\n-- Remove from array (by value)\nUPDATE data.products\nSET attributes = jsonb_set(\n attributes,\n '{tags}',\n (SELECT jsonb_agg(elem) \n FROM jsonb_array_elements(attributes->'tags') AS elem \n WHERE elem != '\"old_tag\"')\n)\nWHERE id = $1;\n\n-- Insert at position\nUPDATE data.products\nSET attributes = jsonb_set(\n attributes,\n '{tags}',\n jsonb_insert(attributes->'tags', '{0}', '\"first_tag\"'::jsonb)\n)\nWHERE id = $1;\n```\n\n### API Function for Partial Updates\n\n```sql\nCREATE PROCEDURE api.update_product_attributes(\n in_product_id uuid,\n in_updates jsonb -- {\"color\": \"red\", \"size\": null} null = delete key\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_key text;\n l_value jsonb;\n l_current jsonb;\nBEGIN\n SELECT attributes INTO l_current\n FROM data.products\n WHERE id = in_product_id\n FOR UPDATE;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Product not found: %', in_product_id\n USING ERRCODE = 'P0002';\n END IF;\n \n -- Process each update\n FOR l_key, l_value IN SELECT * FROM jsonb_each(in_updates)\n LOOP\n IF l_value IS NULL OR l_value = 'null'::jsonb THEN\n -- Remove key\n l_current := l_current - l_key;\n ELSE\n -- Set key\n l_current := jsonb_set(l_current, ARRAY[l_key], l_value);\n END IF;\n END LOOP;\n \n UPDATE data.products\n SET attributes = l_current,\n updated_at = now()\n WHERE id = in_product_id;\nEND;\n$;\n```\n\n## Performance Optimization\n\n### Avoid Large JSONB Documents\n\n```sql\n-- BAD: Entire history in one document (grows unbounded)\nUPDATE data.products\nSET attributes = jsonb_set(\n attributes,\n '{price_history}',\n (attributes->'price_history') || jsonb_build_object('date', now(), 'price', new_price)\n);\n\n-- GOOD: Separate table for history\nCREATE TABLE data.product_price_history (\n product_id uuid REFERENCES data.products(id),\n price numeric(10,2) NOT NULL,\n recorded_at timestamptz NOT NULL DEFAULT now()\n);\n```\n\n### Use JSONB Functions Efficiently\n\n```sql\n-- BAD: Multiple extractions\nSELECT * FROM data.products\nWHERE attributes->>'color' = 'red'\n AND attributes->>'size' = 'large'\n AND attributes->>'material' = 'cotton';\n\n-- GOOD: Single containment check (uses GIN index)\nSELECT * FROM data.products\nWHERE attributes @> '{\"color\": \"red\", \"size\": \"large\", \"material\": \"cotton\"}';\n```\n\n### Denormalize Hot Paths\n\n```sql\n-- If you frequently query by category, add a column\nALTER TABLE data.products ADD COLUMN category text \n GENERATED ALWAYS AS (attributes->>'category') STORED;\n\nCREATE INDEX products_category ON data.products(category);\n\n-- Queries use regular B-tree index\nSELECT * FROM data.products WHERE category = 'electronics';\n```\n\n### TOAST Compression\n\n```sql\n-- JSONB is automatically TOASTed (compressed) for large values\n-- Check TOAST statistics\nSELECT \n relname,\n pg_size_pretty(pg_relation_size(oid)) AS main_size,\n pg_size_pretty(pg_relation_size(reltoastrelid)) AS toast_size\nFROM pg_class\nWHERE relname = 'products';\n\n-- For very large JSONB, consider external storage\n-- ALTER TABLE data.products ALTER COLUMN metadata SET STORAGE EXTERNAL;\n```\n\n### Query Plan Analysis\n\n```sql\n-- Check if GIN index is being used\nEXPLAIN (ANALYZE, BUFFERS)\nSELECT * FROM data.products\nWHERE attributes @> '{\"color\": \"red\"}';\n\n-- Should show: Bitmap Index Scan on products_attributes\n-- If showing Seq Scan, check:\n-- 1. Index exists\n-- 2. Statistics are current (ANALYZE data.products)\n-- 3. Table isn't too small (planner may prefer seq scan)\n```\nd Design (Relational)\n\n```sql\n-- Separate tables for structured data\nCREATE TABLE data.products (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n price numeric(10,2) NOT NULL,\n category_id uuid REFERENCES data.categories(id),\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE TABLE data.product_attributes (\n product_id uuid REFERENCES data.products(id) ON DELETE CASCADE,\n name text NOT NULL,\n value text NOT NULL,\n PRIMARY KEY (product_id, name)\n);\n\n-- Query: Products with specific attribute\nSELECT p.* \nFROM data.products p\nJOIN data.product_attributes pa ON pa.product_id = p.id\nWHERE pa.name = 'color' AND pa.value = 'red';\n```\n\n### JSONB Design (Semi-Structured)\n\n```sql\n-- Single table with flexible attributes\nCREATE TABLE data.products (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n price numeric(10,2) NOT NULL,\n category_id uuid REFERENCES data.categories(id),\n attributes jsonb NOT NULL DEFAULT '{}',\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Query: Products with specific attribute\nSELECT * FROM data.products\nWHERE attributes->>'color' = 'red';\n\n-- Or using containment\nSELECT * FROM data.products\nWHERE attributes @> '{\"color\": \"red\"}';\n```\n\n### Hybrid Design (Best of Both)\n\n```sql\n-- Core fields as columns, extras as JSONB\nCREATE TABLE data.products (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n \n -- Frequently queried: columns\n name text NOT NULL,\n price numeric(10,2) NOT NULL,\n category_id uuid REFERENCES data.categories(id),\n brand text,\n is_active boolean NOT NULL DEFAULT true,\n \n -- Variable/optional: JSONB\n attributes jsonb NOT NULL DEFAULT '{}',\n metadata jsonb NOT NULL DEFAULT '{}',\n \n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Extract common attributes to generated columns\nALTER TABLE data.products ADD COLUMN \n color text GENERATED ALWAYS AS (attributes->>'color') STORED;\n\n-- Now can index and query efficiently\nCREATE INDEX products_color ON data.products(color) WHERE color IS NOT NULL;\n```\n\n## Schema Design with JSONB\n\n### Event/Audit Log Pattern\n\n```sql\nCREATE TABLE data.events (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n event_type text NOT NULL,\n entity_type text NOT NULL,\n entity_id uuid NOT NULL,\n actor_id uuid,\n payload jsonb NOT NULL DEFAULT '{}',\n occurred_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Index on event_type for filtering\nCREATE INDEX events_type_idx ON data.events(event_type);\n\n-- Index on entity for lookup\nCREATE INDEX events_entity_idx ON data.events(entity_type, entity_id);\n\n-- GIN index on payload for flexible queries\nCREATE INDEX events_payload_idx ON data.events USING gin(payload);\n\n-- Example events\nINSERT INTO data.events (event_type, entity_type, entity_id, payload) VALUES\n ('order.created', 'order', 'uuid-1', '{\"total\": 100, \"items\": 3}'),\n ('order.shipped', 'order', 'uuid-1', '{\"carrier\": \"fedex\", \"tracking\": \"123\"}'),\n ('user.login', 'user', 'uuid-2', '{\"ip\": \"1.2.3.4\", \"device\": \"mobile\"}');\n```\n\n### User Preferences Pattern\n\n```sql\nCREATE TABLE data.user_preferences (\n user_id uuid PRIMARY KEY REFERENCES data.users(id),\n preferences jsonb NOT NULL DEFAULT '{\n \"theme\": \"light\",\n \"notifications\": {\n \"email\": true,\n \"push\": true,\n \"sms\": false\n },\n \"language\": \"en\"\n }',\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- API function to get preference\nCREATE FUNCTION api.get_user_preference(in_user_id uuid, in_path text[])\nRETURNS jsonb\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT preferences #> in_path\n FROM data.user_preferences\n WHERE user_id = in_user_id;\n$;\n\n-- Usage: SELECT api.get_user_preference('user-id', ARRAY['notifications', 'email']);\n-- Returns: true\n```\n\n### Form Submissions Pattern\n\n```sql\nCREATE TABLE data.form_submissions (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n form_id uuid NOT NULL REFERENCES data.forms(id),\n submitter_id uuid,\n data jsonb NOT NULL,\n submitted_at timestamptz NOT NULL DEFAULT now(),\n \n -- Extract common fields for indexing/querying\n email text GENERATED ALWAYS AS (data->>'email') STORED,\n status text GENERATED ALWAYS AS (COALESCE(data->>'status', 'pending')) STORED\n);\n\nCREATE INDEX submissions_email ON data.form_submissions(email) WHERE email IS NOT NULL;\nCREATE INDEX submissions_status ON data.form_submissions(status);\n```\n\n## Indexing Strategies\n\n### GIN Index (Default)\n\n```sql\n-- General purpose: indexes all keys and values\nCREATE INDEX products_attrs ON data.products USING gin(attributes);\n\n-- Supports operators: @>, ?, ?&, ?|, @?\nSELECT * FROM data.products WHERE attributes @> '{\"color\": \"red\"}';\nSELECT * FROM data.products WHERE attributes ? 'warranty';\nSELECT * FROM data.products WHERE attributes ?& array['color', 'size'];\n```\n\n### GIN with jsonb_path_ops (Faster @>)\n\n```sql\n-- Optimized for containment queries only\nCREATE INDEX products_attrs_path \n ON data.products USING gin(attributes jsonb_path_ops);\n\n-- Only supports @> operator, but faster and smaller\nSELECT * FROM data.products WHERE attributes @> '{\"category\": \"electronics\"}';\n```\n\n### B-tree on Extracted Value\n\n```sql\n-- For specific key queries\nCREATE INDEX products_color \n ON data.products((attributes->>'color'));\n\n-- Or using generated column (PostgreSQL 12+)\nALTER TABLE data.products ADD COLUMN \n color text GENERATED ALWAYS AS (attributes->>'color') STORED;\nCREATE INDEX products_color ON data.products(color);\n\n-- Much faster for: WHERE attributes->>'color' = 'red'\n```\n\n### Expression Index for Nested Values\n\n```sql\n-- Index nested path\nCREATE INDEX products_brand \n ON data.products((attributes->'specs'->>'brand'));\n\n-- Query uses index\nSELECT * FROM data.products \nWHERE attributes->'specs'->>'brand' = 'Apple';\n```\n\n### Index Selection Guide\n\n| Query Pattern | Index Type | Example |\n|---------------|------------|---------|\n| `@>` containment | GIN or GIN jsonb_path_ops | `WHERE data @> '{\"key\": \"value\"}'` |\n| `?` key exists | GIN | `WHERE data ? 'key'` |\n| `->>` specific key equality | B-tree expression | `WHERE data->>'key' = 'value'` |\n| `->>` specific key range | B-tree expression | `WHERE (data->>'count')::int > 10` |\n| Full-text in values | GIN with to_tsvector | Custom |\n\n## Query Patterns\n\n### Basic Access\n\n```sql\n-- Get value as JSONB\nSELECT attributes->'color' FROM data.products; -- Returns: \"red\" (with quotes)\n\n-- Get value as text\nSELECT attributes->>'color' FROM data.products; -- Returns: red (no quotes)\n\n-- Nested access\nSELECT attributes->'specs'->'dimensions'->>'width' FROM data.products;\n\n-- Path access (array of keys)\nSELECT attributes #> '{specs,dimensions,width}' FROM data.products;\nSELECT attributes #>> '{specs,dimensions,width}' FROM data.products; -- As text\n```\n\n### Filtering\n\n```sql\n-- Containment: JSONB contains this structure\nSELECT * FROM data.products\nWHERE attributes @> '{\"color\": \"red\", \"size\": \"large\"}';\n\n-- Key exists\nSELECT * FROM data.products WHERE attributes ? 'warranty';\n\n-- Any key exists\nSELECT * FROM data.products WHERE attributes ?| array['warranty', 'guarantee'];\n\n-- All keys exist\nSELECT * FROM data.products WHERE attributes ?& array['color', 'size'];\n\n-- Value comparison\nSELECT * FROM data.products WHERE attributes->>'color' = 'red';\nSELECT * FROM data.products WHERE (attributes->>'price')::numeric > 100;\n\n-- JSON path query (PostgreSQL 12+)\nSELECT * FROM data.products\nWHERE attributes @? '$.specs.dimensions ? (@.width > 10)';\n```\n\n### Aggregation\n\n```sql\n-- Count by JSONB value\nSELECT \n attributes->>'color' AS color,\n COUNT(*) AS count\nFROM data.products\nWHERE attributes ? 'color'\nGROUP BY attributes->>'color';\n\n-- Sum numeric values in JSONB\nSELECT SUM((item->>'quantity')::int * (item->>'price')::numeric)\nFROM data.orders,\n jsonb_array_elements(items) AS item;\n\n-- Distinct values\nSELECT DISTINCT attributes->>'color' AS color\nFROM data.products\nWHERE attributes ? 'color';\n```\n\n### Array Operations\n\n```sql\n-- JSONB array contains value\nSELECT * FROM data.products\nWHERE attributes->'tags' ? 'sale';\n\n-- Expand array elements\nSELECT \n p.id,\n p.name,\n tag.value AS tag\nFROM data.products p,\n jsonb_array_elements_text(p.attributes->'tags') AS tag(value);\n\n-- Array length\nSELECT * FROM data.products\nWHERE jsonb_array_length(attributes->'tags') > 3;\n\n-- Check if array contains\nSELECT * FROM data.products\nWHERE attributes->'tags' @> '[\"electronics\", \"sale\"]';\n```\n\n### Joining JSONB Data\n\n```sql\n-- Join on JSONB extracted value\nSELECT \n o.id,\n o.data->>'customer_email' AS email,\n c.name\nFROM data.orders o\nJOIN data.customers c ON c.email = o.data->>'customer_email';\n\n-- Better: Extract to column if frequently joined\nALTER TABLE data.orders ADD COLUMN \n customer_email text GENERATED ALWAYS AS (data->>'customer_email') STORED;\nCREATE INDEX orders_customer_email ON data.orders(customer_email);\n```\n\n## Validation Patterns\n\n### CHECK Constraint Validation\n\n```sql\n-- Ensure required keys exist\nALTER TABLE data.products ADD CONSTRAINT products_attrs_required\n CHECK (\n attributes ? 'name'\n AND attributes ? 'sku'\n );\n\n-- Validate value types\nALTER TABLE data.products ADD CONSTRAINT products_attrs_types\n CHECK (\n jsonb_typeof(attributes->'price') = 'number'\n AND jsonb_typeof(attributes->'tags') = 'array'\n );\n\n-- Validate enum values\nALTER TABLE data.orders ADD CONSTRAINT orders_status_valid\n CHECK (\n data->>'status' IN ('draft', 'pending', 'confirmed', 'shipped', 'delivered')\n );\n\n-- Combined validation\nALTER TABLE data.form_submissions ADD CONSTRAINT form_data_valid\n CHECK (\n -- Required fields\n data ? 'email'\n AND data ? 'name'\n -- Email format (basic)\n AND data->>'email' ~ '^[^@]+@[^@]+\\.[^@]+

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

\n -- Name length\n AND length(data->>'name') BETWEEN 1 AND 100\n );\n```\n\n### Validation Function\n\n```sql\nCREATE OR REPLACE FUNCTION private.validate_product_attributes(in_attrs jsonb)\nRETURNS boolean\nLANGUAGE plpgsql\nIMMUTABLE\nAS $\nDECLARE\n l_errors text[] := '{}';\nBEGIN\n -- Required fields\n IF NOT (in_attrs ? 'sku') THEN\n l_errors := array_append(l_errors, 'sku is required');\n END IF;\n \n -- Type validation\n IF in_attrs ? 'price' AND jsonb_typeof(in_attrs->'price') != 'number' THEN\n l_errors := array_append(l_errors, 'price must be a number');\n END IF;\n \n -- Range validation\n IF in_attrs ? 'price' AND (in_attrs->>'price')::numeric \u003c 0 THEN\n l_errors := array_append(l_errors, 'price must be non-negative');\n END IF;\n \n -- Tags validation\n IF in_attrs ? 'tags' THEN\n IF jsonb_typeof(in_attrs->'tags') != 'array' THEN\n l_errors := array_append(l_errors, 'tags must be an array');\n ELSIF jsonb_array_length(in_attrs->'tags') > 10 THEN\n l_errors := array_append(l_errors, 'maximum 10 tags allowed');\n END IF;\n END IF;\n \n IF array_length(l_errors, 1) > 0 THEN\n RAISE EXCEPTION 'Validation failed: %', array_to_string(l_errors, ', ')\n USING ERRCODE = 'P0001';\n END IF;\n \n RETURN true;\nEND;\n$;\n\n-- Use in CHECK constraint\nALTER TABLE data.products ADD CONSTRAINT products_attrs_valid\n CHECK (private.validate_product_attributes(attributes));\n```\n\n## Update Patterns\n\n### Set Single Key\n\n```sql\n-- Set/replace a key\nUPDATE data.products\nSET attributes = jsonb_set(attributes, '{color}', '\"blue\"')\nWHERE id = 'product-uuid';\n\n-- Set nested key (creates path if needed)\nUPDATE data.products\nSET attributes = jsonb_set(attributes, '{specs,weight}', '2.5')\nWHERE id = 'product-uuid';\n\n-- Set with create_if_missing = true (default)\nUPDATE data.products\nSET attributes = jsonb_set(attributes, '{new_key}', '\"value\"', true)\nWHERE id = 'product-uuid';\n```\n\n### Remove Key\n\n```sql\n-- Remove single key\nUPDATE data.products\nSET attributes = attributes - 'color'\nWHERE id = 'product-uuid';\n\n-- Remove nested key\nUPDATE data.products\nSET attributes = attributes #- '{specs,weight}'\nWHERE id = 'product-uuid';\n\n-- Remove multiple keys\nUPDATE data.products\nSET attributes = attributes - 'color' - 'size'\nWHERE id = 'product-uuid';\n```\n\n### Merge/Concat\n\n```sql\n-- Merge objects (right side wins on conflict)\nUPDATE data.products\nSET attributes = attributes || '{\"color\": \"green\", \"size\": \"XL\"}'\nWHERE id = 'product-uuid';\n\n-- Deep merge function\nCREATE OR REPLACE FUNCTION private.jsonb_merge_deep(in_a jsonb, in_b jsonb)\nRETURNS jsonb\nLANGUAGE sql\nIMMUTABLE\nAS $\n SELECT \n CASE\n WHEN jsonb_typeof(in_a) = 'object' AND jsonb_typeof(in_b) = 'object' THEN\n (SELECT jsonb_object_agg(\n COALESCE(ka, kb),\n CASE\n WHEN va IS NULL THEN vb\n WHEN vb IS NULL THEN va\n ELSE private.jsonb_merge_deep(va, vb)\n END\n )\n FROM jsonb_each(in_a) AS a(ka, va)\n FULL JOIN jsonb_each(in_b) AS b(kb, vb) ON ka = kb)\n ELSE in_b\n END;\n$;\n```\n\n### Array Modifications\n\n```sql\n-- Append to array\nUPDATE data.products\nSET attributes = jsonb_set(\n attributes,\n '{tags}',\n COALESCE(attributes->'tags', '[]'::jsonb) || '\"new_tag\"'\n)\nWHERE id = 'product-uuid';\n\n-- Remove from array (by value)\nUPDATE data.products\nSET attributes = jsonb_set(\n attributes,\n '{tags}',\n (SELECT jsonb_agg(elem) \n FROM jsonb_array_elements(attributes->'tags') elem \n WHERE elem != '\"remove_this\"')\n)\nWHERE id = 'product-uuid';\n```\n\n### API Function for Updates\n\n```sql\nCREATE PROCEDURE api.update_product_attributes(\n in_product_id uuid,\n in_updates jsonb\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_current jsonb;\nBEGIN\n -- Get current attributes\n SELECT attributes INTO l_current\n FROM data.products\n WHERE id = in_product_id\n FOR UPDATE;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Product not found: %', in_product_id\n USING ERRCODE = 'P0002';\n END IF;\n \n -- Merge updates\n UPDATE data.products\n SET attributes = l_current || in_updates\n WHERE id = in_product_id;\nEND;\n$;\n```\n\n## Performance Optimization\n\n### Avoid Full Document Reads\n\n```sql\n-- BAD: Fetches entire JSONB then extracts\nSELECT attributes->>'name' FROM data.products;\n\n-- BETTER: Use covering index\nCREATE INDEX products_attrs_name \n ON data.products((attributes->>'name')) \n INCLUDE (id);\n\n-- Query can use index-only scan\nSELECT id, attributes->>'name' FROM data.products;\n```\n\n### Partial Index on JSONB\n\n```sql\n-- Index only documents with specific type\nCREATE INDEX events_order_payload_idx \n ON data.events USING gin(payload)\n WHERE event_type LIKE 'order.%';\n\n-- Query automatically uses partial index\nSELECT * FROM data.events\nWHERE event_type = 'order.created'\n AND payload @> '{\"total\": 100}';\n```\n\n### Benchmark Queries\n\n```sql\n-- Compare query performance\nEXPLAIN (ANALYZE, BUFFERS)\nSELECT * FROM data.products WHERE attributes @> '{\"color\": \"red\"}';\n\nEXPLAIN (ANALYZE, BUFFERS)\nSELECT * FROM data.products WHERE attributes->>'color' = 'red';\n\n-- The @> containment operator uses GIN index\n-- The ->> extraction might use B-tree expression index\n-- Test both with your actual data distribution\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":34291,"content_sha256":"b3dda8c7a42255f562965734df84f7eb9e2922434bce426d7cb495a03d90ec63"},{"filename":"references/migrations.md","content":"# Database Migrations System\n\n## Table of Contents\n1. [Overview](#overview)\n2. [Migration Schema Setup](#migration-schema-setup)\n3. [Migration Log Table](#migration-log-table)\n4. [Locking Mechanism](#locking-mechanism)\n5. [Checksum Validation](#checksum-validation)\n6. [Migration Types](#migration-types)\n7. [Core Migration Functions](#core-migration-functions)\n8. [Migration File Conventions](#migration-file-conventions)\n9. [Rollback Support](#rollback-support)\n10. [Idempotent Constraint Creation](#idempotent-constraint-creation)\n11. [Blue-Green Deployment Patterns](#blue-green-deployment-patterns)\n12. [Complete Implementation](#complete-implementation)\n\n## Overview\n\nA pure PL/pgSQL migration system that provides:\n- Version-controlled schema changes\n- Concurrent execution protection via advisory locks\n- Checksum validation to detect modified migrations\n- Versioned and repeatable migration support\n- Rollback capability\n- Full execution history\n\n### System Architecture\n\n```mermaid\nflowchart TB\n subgraph RUNNER[\"Migration Runner\"]\n CLI[\"psql / Application\"]\n end\n \n subgraph MIGRATION_SYSTEM[\"app_migration Schema\"]\n LOCK[\"Advisory Lock\u003cbr/>acquire_lock()\u003cbr/>release_lock()\"]\n EXEC[\"Executor\u003cbr/>run_versioned()\u003cbr/>run_repeatable()\"]\n VALID[\"Validator\u003cbr/>calculate_checksum()\u003cbr/>is_version_applied()\"]\n LOG[(\"changelog\u003cbr/>table\")]\n ROLLBACK_T[(\"rollback_scripts\u003cbr/>table\")]\n end\n \n subgraph TARGET[\"Target Schemas\"]\n DATA[(\"data\")]\n PRIVATE[(\"private\")]\n API[(\"api\")]\n end\n \n CLI -->|\"1. acquire_lock()\"| LOCK\n CLI -->|\"2. run_versioned()\"| EXEC\n EXEC -->|\"3. validate\"| VALID\n VALID -->|\"4. check\"| LOG\n EXEC -->|\"5. execute SQL\"| TARGET\n EXEC -->|\"6. record\"| LOG\n CLI -->|\"7. release_lock()\"| LOCK\n \n style MIGRATION_SYSTEM fill:#e3f2fd\n style TARGET fill:#c8e6c9\n```\n\n### Migration Execution Flow\n\n```mermaid\nsequenceDiagram\n participant R as Runner\n participant L as Lock System\n participant V as Validator\n participant E as Executor\n participant C as Changelog\n \n R->>L: acquire_lock()\n alt Lock acquired\n L-->>R: true\n R->>V: is_version_applied('001')\n alt Not applied\n V-->>R: false\n R->>E: execute(migration_sql)\n E->>E: Run DDL/DML\n E->>C: INSERT changelog record\n E-->>R: success\n else Already applied\n V-->>R: true\n V->>V: verify_checksum()\n alt Checksum matches\n V-->>R: skip (already applied)\n else Checksum mismatch\n V-->>R: ERROR: modified!\n end\n end\n R->>L: release_lock()\n else Lock not available\n L-->>R: false (another migration running)\n end\n```\n\n## Migration Schema Setup\n\n```sql\n-- Create dedicated schema for migration system\nCREATE SCHEMA IF NOT EXISTS app_migration;\n\nCOMMENT ON SCHEMA app_migration IS 'Database migration management system';\n```\n\n## Migration Log Table\n\nTrack all executed migrations:\n\n```mermaid\nerDiagram\n CHANGELOG {\n bigint id PK \"GENERATED ALWAYS AS IDENTITY\"\n text version \"Migration version (001, 002, etc.)\"\n text description \"Human-readable description\"\n text type \"versioned|repeatable|baseline\"\n text filename \"Original filename\"\n text checksum \"MD5 of normalized content\"\n integer execution_time_ms \"Execution duration\"\n timestamptz executed_at \"When it ran\"\n text executed_by \"current_user\"\n boolean success \"true if successful\"\n }\n \n ROLLBACK_SCRIPTS {\n text version PK \"Matches changelog version\"\n text rollback_sql \"SQL to undo migration\"\n timestamptz created_at\n text created_by\n }\n \n ROLLBACK_HISTORY {\n bigint id PK\n bigint changelog_id FK\n text version\n text rollback_sql\n timestamptz rolled_back_at\n text rolled_back_by\n boolean success\n }\n \n CHANGELOG ||--o| ROLLBACK_SCRIPTS : \"has optional\"\n CHANGELOG ||--o{ ROLLBACK_HISTORY : \"tracks\"\n```\n\n```sql\nCREATE TABLE app_migration.changelog (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n version text NOT NULL,\n description text NOT NULL,\n type text NOT NULL DEFAULT 'versioned', -- 'versioned', 'repeatable', 'baseline'\n filename text NOT NULL,\n checksum text NOT NULL,\n execution_time_ms integer,\n executed_at timestamptz NOT NULL DEFAULT now(),\n executed_by text NOT NULL DEFAULT current_user,\n success boolean NOT NULL DEFAULT true,\n \n CONSTRAINT changelog_type_check \n CHECK (type IN ('versioned', 'repeatable', 'baseline'))\n);\n\n-- Unique constraint for versioned migrations\nCREATE UNIQUE INDEX changelog_version_key \n ON app_migration.changelog(version) \n WHERE type = 'versioned';\n\n-- Index for ordering\nCREATE INDEX changelog_executed_idx ON app_migration.changelog(executed_at);\n\nCOMMENT ON TABLE app_migration.changelog IS 'Records all migration executions';\n```\n\n## Locking Mechanism\n\n### Advisory Lock Flow\n\n```mermaid\nflowchart TD\n START([Migration Start]) --> TRY_LOCK{pg_try_advisory_lock}\n \n TRY_LOCK -->|\"Returns TRUE\"| LOCKED[\"Lock Acquired ✓\"]\n TRY_LOCK -->|\"Returns FALSE\"| NOT_LOCKED[\"Lock Held by Another\"]\n \n LOCKED --> RUN_MIG[\"Run Migrations\"]\n NOT_LOCKED --> WAIT{Wait with timeout?}\n \n WAIT -->|\"Yes\"| RETRY[\"Sleep & Retry\"]\n WAIT -->|\"No / Timeout\"| FAIL[\"Fail: Lock unavailable\"]\n \n RETRY --> TRY_LOCK\n \n RUN_MIG --> RELEASE[\"pg_advisory_unlock\"]\n RELEASE --> DONE([Done])\n FAIL --> DONE\n \n style LOCKED fill:#c8e6c9\n style NOT_LOCKED fill:#ffcdd2\n style FAIL fill:#ffcdd2\n```\n\nUse PostgreSQL advisory locks to prevent concurrent migrations:\n\n```sql\n-- Lock configuration\nCREATE TABLE app_migration.lock_config (\n id integer PRIMARY KEY DEFAULT 1,\n lock_id bigint NOT NULL DEFAULT 123456789, -- Advisory lock identifier\n \n CONSTRAINT lock_config_single_row CHECK (id = 1)\n);\n\nINSERT INTO app_migration.lock_config DEFAULT VALUES;\n\n-- Acquire migration lock\nCREATE FUNCTION app_migration.acquire_lock()\nRETURNS boolean\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_lock_id bigint;\n l_acquired boolean;\nBEGIN\n SELECT lock_id INTO l_lock_id FROM app_migration.lock_config;\n\n -- Try to acquire advisory lock (non-blocking)\n l_acquired := pg_try_advisory_lock(l_lock_id);\n\n IF NOT l_acquired THEN\n RAISE NOTICE 'Migration lock not available - another migration is running';\n END IF;\n\n RETURN l_acquired;\nEND;\n$;\n\n-- Acquire lock with wait\nCREATE FUNCTION app_migration.acquire_lock_wait(in_timeout_seconds integer DEFAULT 30)\nRETURNS boolean\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_lock_id bigint;\n l_start_time timestamptz := clock_timestamp();\nBEGIN\n SELECT lock_id INTO l_lock_id FROM app_migration.lock_config;\n\n -- Try to acquire, with timeout\n LOOP\n IF pg_try_advisory_lock(l_lock_id) THEN\n RETURN true;\n END IF;\n\n IF clock_timestamp() > l_start_time + make_interval(secs := in_timeout_seconds) THEN\n RAISE EXCEPTION 'Timeout acquiring migration lock after % seconds', in_timeout_seconds;\n END IF;\n\n -- Wait 100ms before retry\n PERFORM pg_sleep(0.1);\n END LOOP;\nEND;\n$;\n\n-- Release migration lock\nCREATE FUNCTION app_migration.release_lock()\nRETURNS boolean\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_lock_id bigint;\nBEGIN\n SELECT lock_id INTO l_lock_id FROM app_migration.lock_config;\n RETURN pg_advisory_unlock(l_lock_id);\nEND;\n$;\n\n-- Check if lock is held\nCREATE FUNCTION app_migration.is_locked()\nRETURNS boolean\nLANGUAGE sql\nSTABLE\nAS $\n SELECT EXISTS (\n SELECT 1 FROM pg_locks \n WHERE locktype = 'advisory' \n AND classid = (SELECT (lock_id >> 32)::integer FROM app_migration.lock_config)\n AND objid = (SELECT (lock_id & x'FFFFFFFF'::bigint)::integer FROM app_migration.lock_config)\n );\n$;\n```\n\n## Checksum Validation\n\nDetect if migrations have been modified after execution:\n\n```sql\n-- Calculate checksum for migration content\nCREATE FUNCTION app_migration.calculate_checksum(in_content text)\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nPARALLEL SAFE\nAS $\n -- Strip whitespace normalization and compute MD5\n SELECT md5(\n regexp_replace(\n regexp_replace(in_content, '\\s+', ' ', 'g'), -- Normalize whitespace\n '^\\s+|\\s+

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

, '', 'g' -- Trim\n )\n );\n$;\n\n-- Validate all executed migration checksums\nCREATE FUNCTION app_migration.validate_checksums()\nRETURNS TABLE (\n version text,\n filename text,\n status text,\n stored_checksum text,\n current_checksum text\n)\nLANGUAGE plpgsql\nSTABLE\nAS $\nBEGIN\n -- This function should be called after loading migration files\n -- Implementation depends on how migrations are stored/accessed\n -- Return empty set as placeholder - actual validation happens during run\n RETURN;\nEND;\n$;\n```\n\n## Migration Types\n\n### Versioned Migrations\n\nApplied exactly once, in order:\n\n```sql\n-- Migration file naming: V{version}__{description}.sql\n-- Example: V001__create_users_table.sql\n\n-- Check if version already applied\nCREATE FUNCTION app_migration.is_version_applied(in_version text)\nRETURNS boolean\nLANGUAGE sql\nSTABLE\nAS $\n SELECT EXISTS (\n SELECT 1 FROM app_migration.changelog \n WHERE version = in_version \n AND type = 'versioned' \n AND success = true\n );\n$;\n```\n\n### Repeatable Migrations\n\nRe-applied whenever content changes:\n\n```sql\n-- Migration file naming: R__{description}.sql\n-- Example: R__views.sql\n\n-- Get stored checksum for repeatable migration\nCREATE FUNCTION app_migration.get_repeatable_checksum(in_filename text)\nRETURNS text\nLANGUAGE sql\nSTABLE\nAS $\n SELECT checksum\n FROM app_migration.changelog\n WHERE filename = in_filename\n AND type = 'repeatable'\n AND success = true\n ORDER BY executed_at DESC\n LIMIT 1;\n$;\n\n-- Check if repeatable needs to run\nCREATE FUNCTION app_migration.repeatable_needs_run(\n in_filename text,\n in_content text\n)\nRETURNS boolean\nLANGUAGE sql\nSTABLE\nAS $\n SELECT COALESCE(\n app_migration.get_repeatable_checksum(in_filename) \n != app_migration.calculate_checksum(in_content),\n true -- Never run before\n );\n$;\n```\n\n### Baseline\n\nMark existing database as starting point:\n\n```sql\n-- Set baseline version\nCREATE PROCEDURE app_migration.set_baseline(\n in_version text,\n in_description text DEFAULT 'Baseline'\n)\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Clear any existing changelog\n DELETE FROM app_migration.changelog;\n \n -- Insert baseline marker\n INSERT INTO app_migration.changelog (\n version, description, type, filename, checksum\n ) VALUES (\n in_version, \n in_description, \n 'baseline', \n 'BASELINE',\n 'BASELINE'\n );\n \n RAISE NOTICE 'Baseline set to version %', in_version;\nEND;\n$;\n```\n\n## Core Migration Functions\n\n### Get Current Version\n\n```sql\nCREATE FUNCTION app_migration.get_current_version()\nRETURNS text\nLANGUAGE sql\nSTABLE\nAS $\n SELECT COALESCE(\n (SELECT version \n FROM app_migration.changelog \n WHERE type IN ('versioned', 'baseline') \n AND success = true\n ORDER BY executed_at DESC \n LIMIT 1),\n '0'\n );\n$;\n```\n\n### Register Migration Execution\n\n```sql\nCREATE PROCEDURE app_migration.register_migration(\n in_version text,\n in_description text,\n in_type text,\n in_filename text,\n in_checksum text,\n in_execution_time_ms integer DEFAULT NULL,\n in_success boolean DEFAULT true\n)\nLANGUAGE plpgsql\nAS $\nBEGIN\n INSERT INTO app_migration.changelog (\n version, description, type, filename, checksum, \n execution_time_ms, success\n ) VALUES (\n in_version, in_description, in_type, in_filename,\n in_checksum, in_execution_time_ms, in_success\n );\nEND;\n$;\n```\n\n### Execute Single Migration\n\n```sql\nCREATE PROCEDURE app_migration.execute_migration(\n in_version text,\n in_description text,\n in_type text,\n in_filename text,\n in_sql text,\n in_validate_checksum boolean DEFAULT true\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_checksum text;\n l_stored_checksum text;\n l_start_time timestamptz;\n l_execution_time_ms integer;\nBEGIN\n -- Calculate checksum\n l_checksum := app_migration.calculate_checksum(in_sql);\n\n -- For versioned migrations, check if already applied\n IF in_type = 'versioned' THEN\n IF app_migration.is_version_applied(in_version) THEN\n -- Verify checksum hasn't changed\n SELECT checksum INTO l_stored_checksum\n FROM app_migration.changelog\n WHERE version = in_version AND type = 'versioned' AND success = true;\n\n IF in_validate_checksum AND l_stored_checksum != l_checksum THEN\n RAISE EXCEPTION 'Checksum mismatch for version %: stored=%, current=%',\n in_version, l_stored_checksum, l_checksum;\n END IF;\n\n RAISE NOTICE 'Version % already applied, skipping', in_version;\n RETURN;\n END IF;\n END IF;\n\n -- For repeatable migrations, check if content changed\n IF in_type = 'repeatable' THEN\n IF NOT app_migration.repeatable_needs_run(in_filename, in_sql) THEN\n RAISE NOTICE 'Repeatable migration % unchanged, skipping', in_filename;\n RETURN;\n END IF;\n END IF;\n\n -- Execute migration\n l_start_time := clock_timestamp();\n\n BEGIN\n EXECUTE in_sql;\n\n l_execution_time_ms := extract(milliseconds from clock_timestamp() - l_start_time)::integer;\n\n -- Register successful execution\n CALL app_migration.register_migration(\n in_version, in_description, in_type, in_filename,\n l_checksum, l_execution_time_ms, true\n );\n\n RAISE NOTICE 'Applied % migration: % (% ms)',\n in_type, in_version, l_execution_time_ms;\n\n EXCEPTION WHEN OTHERS THEN\n -- Register failed execution\n CALL app_migration.register_migration(\n in_version, in_description, in_type, in_filename,\n l_checksum, NULL, false\n );\n\n RAISE EXCEPTION 'Migration % failed: %', in_version, SQLERRM;\n END;\nEND;\n$;\n```\n\n### Get Pending Migrations\n\n```sql\nCREATE FUNCTION app_migration.get_pending(in_versions text[] DEFAULT NULL)\nRETURNS TABLE (\n version text,\n is_new boolean\n)\nLANGUAGE sql\nSTABLE\nAS $\n -- Return versions from input array that haven't been applied\n -- Implementation depends on how versions are provided\n SELECT \n unnest(in_versions),\n true\n WHERE unnest(in_versions) NOT IN (\n SELECT version \n FROM app_migration.changelog \n WHERE type = 'versioned' AND success = true\n );\n$;\n```\n\n### Get Migration History\n\n```sql\nCREATE FUNCTION app_migration.get_history(in_limit integer DEFAULT 50)\nRETURNS TABLE (\n id bigint,\n version text,\n description text,\n type text,\n filename text,\n checksum text,\n execution_time_ms integer,\n executed_at timestamptz,\n executed_by text,\n success boolean\n)\nLANGUAGE sql\nSTABLE\nAS $\n SELECT \n id, version, description, type, filename, checksum,\n execution_time_ms, executed_at, executed_by, success\n FROM app_migration.changelog\n ORDER BY executed_at DESC\n LIMIT in_limit;\n$;\n```\n\n### Migration Info (Summary)\n\n```sql\nCREATE FUNCTION app_migration.info()\nRETURNS TABLE (\n current_version text,\n total_migrations bigint,\n last_migration_at timestamptz,\n is_locked boolean\n)\nLANGUAGE sql\nSTABLE\nAS $\n SELECT\n app_migration.get_current_version(),\n (SELECT count(*) FROM app_migration.changelog WHERE success = true),\n (SELECT max(executed_at) FROM app_migration.changelog WHERE success = true),\n app_migration.is_locked();\n$;\n```\n\n## Migration File Conventions\n\n### Versioned Migration Files\n\n```\nmigrations/\n├── V001__initial_schema.sql\n├── V002__create_users_table.sql\n├── V003__add_orders_table.sql\n├── V004__add_user_email_index.sql\n└── ...\n```\n\n**Naming Pattern**: `V{version}__{description}.sql`\n- Version: Zero-padded number (001, 002, ...) or timestamp (20240101120000)\n- Double underscore separator\n- Description: snake_case, no special characters\n\n### Repeatable Migration Files\n\n```\nmigrations/\n├── R__functions.sql\n├── R__views.sql\n├── R__triggers.sql\n└── R__permissions.sql\n```\n\n**Naming Pattern**: `R__{description}.sql`\n- No version number\n- Re-run whenever content changes\n- Applied after all versioned migrations\n\n### Example Versioned Migration\n\n```sql\n-- V001__initial_schema.sql\n-- Description: Create initial application schema and base tables\n-- Author: team\n-- Date: 2024-01-15\n\n-- Create schemas\nCREATE SCHEMA IF NOT EXISTS app;\nCREATE SCHEMA IF NOT EXISTS app_audit;\n\n-- Create customers table\nCREATE TABLE data.customers (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n email text NOT NULL,\n name text NOT NULL,\n is_active boolean NOT NULL DEFAULT true,\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE UNIQUE INDEX customers_email_key ON data.customers(lower(email));\nCREATE INDEX customers_is_active_idx ON data.customers(is_active) WHERE is_active = true;\n\nCOMMENT ON TABLE data.customers IS 'Customer accounts';\n```\n\n### Example Repeatable Migration\n\n```sql\n-- R__views.sql\n-- Description: Application views (re-applied on change)\n\n-- Drop and recreate views to handle definition changes\nDROP VIEW IF EXISTS api.v_active_customers CASCADE;\n\nCREATE VIEW api.v_active_customers AS\nSELECT id, email, name, created_at\nFROM data.customers\nWHERE is_active = true;\n\nDROP VIEW IF EXISTS api.v_order_summary CASCADE;\n\nCREATE VIEW api.v_order_summary AS\nSELECT \n o.id,\n o.customer_id,\n c.email AS customer_email,\n o.status,\n o.total,\n o.created_at\nFROM data.orders o\nJOIN data.customers c ON c.id = o.customer_id;\n```\n\n## Rollback Support\n\n### Rollback History Table\n\n```sql\nCREATE TABLE app_migration.rollback_log (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n changelog_id bigint NOT NULL REFERENCES app_migration.changelog(id),\n version text NOT NULL,\n rollback_sql text,\n rolled_back_at timestamptz NOT NULL DEFAULT now(),\n rolled_back_by text NOT NULL DEFAULT current_user,\n success boolean NOT NULL DEFAULT true\n);\n```\n\n### Store Rollback SQL\n\n```sql\n-- Optional: Store rollback SQL with migration\nCREATE TABLE app_migration.rollback_scripts (\n version text PRIMARY KEY,\n rollback_sql text NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Register rollback script\nCREATE PROCEDURE app_migration.register_rollback(\n in_version text,\n in_rollback_sql text\n)\nLANGUAGE sql\nAS $\n INSERT INTO app_migration.rollback_scripts (version, rollback_sql)\n VALUES (in_version, in_rollback_sql)\n ON CONFLICT (version) DO UPDATE SET rollback_sql = EXCLUDED.rollback_sql;\n$;\n```\n\n### Execute Rollback\n\n```sql\nCREATE PROCEDURE app_migration.rollback_version(in_version text)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_rollback_sql text;\n l_changelog_id bigint;\nBEGIN\n -- Get changelog entry\n SELECT id INTO l_changelog_id\n FROM app_migration.changelog\n WHERE version = in_version AND type = 'versioned' AND success = true\n ORDER BY executed_at DESC\n LIMIT 1;\n\n IF l_changelog_id IS NULL THEN\n RAISE EXCEPTION 'Version % not found in changelog', in_version;\n END IF;\n\n -- Get rollback SQL\n SELECT rollback_sql INTO l_rollback_sql\n FROM app_migration.rollback_scripts\n WHERE version = in_version;\n\n IF l_rollback_sql IS NULL THEN\n RAISE EXCEPTION 'No rollback script for version %', in_version;\n END IF;\n\n -- Execute rollback\n BEGIN\n EXECUTE l_rollback_sql;\n\n -- Log successful rollback\n INSERT INTO app_migration.rollback_log (changelog_id, version, rollback_sql, success)\n VALUES (l_changelog_id, in_version, l_rollback_sql, true);\n\n -- Mark original migration as rolled back (keep in history)\n UPDATE app_migration.changelog\n SET success = false\n WHERE id = l_changelog_id;\n\n RAISE NOTICE 'Rolled back version %', in_version;\n\n EXCEPTION WHEN OTHERS THEN\n -- Log failed rollback\n INSERT INTO app_migration.rollback_log (changelog_id, version, rollback_sql, success)\n VALUES (l_changelog_id, in_version, l_rollback_sql, false);\n\n RAISE EXCEPTION 'Rollback of version % failed: %', in_version, SQLERRM;\n END;\nEND;\n$;\n```\n\n## Idempotent Constraint Creation\n\nPostgreSQL does not support `ADD CONSTRAINT IF NOT EXISTS`. Running `ALTER TABLE ... ADD CONSTRAINT` in a migration that may be re-executed will fail on the second run. Use a `DO` block to check `pg_constraint` first.\n\n### Unique Constraints\n\n```sql\nDO $\nBEGIN\n IF NOT EXISTS (\n SELECT 1 FROM pg_constraint\n WHERE conname = 'customers_email_uk'\n AND conrelid = 'data.customers'::regclass\n ) THEN\n ALTER TABLE data.customers\n ADD CONSTRAINT customers_email_uk UNIQUE (email);\n END IF;\nEND $;\n```\n\n### Foreign Keys\n\n```sql\nDO $\nBEGIN\n IF NOT EXISTS (\n SELECT 1 FROM pg_constraint\n WHERE conname = 'orders_customers_fk'\n AND conrelid = 'data.orders'::regclass\n ) THEN\n ALTER TABLE data.orders\n ADD CONSTRAINT orders_customers_fk\n FOREIGN KEY (customer_id) REFERENCES data.customers(id);\n END IF;\nEND $;\n```\n\n### Check Constraints\n\n```sql\nDO $\nBEGIN\n IF NOT EXISTS (\n SELECT 1 FROM pg_constraint\n WHERE conname = 'orders_total_ck'\n ) THEN\n ALTER TABLE data.orders\n ADD CONSTRAINT orders_total_ck CHECK (total >= 0);\n END IF;\nEND $;\n```\n\n### Inspect Existing Constraints\n\n```sql\n-- List all constraints on a table\nSELECT conname, contype, pg_get_constraintdef(oid)\nFROM pg_constraint\nWHERE conrelid = 'data.orders'::regclass;\n\n-- contype values: 'p' = PRIMARY KEY, 'f' = FOREIGN KEY, 'u' = UNIQUE, 'c' = CHECK\n```\n\n## Blue-Green Deployment Patterns\n\n### Overview\n\nBlue-green deployment for database schema changes ensures zero-downtime migrations by maintaining backward compatibility during transitions.\n\n### Key Principles\n\n```markdown\n1. **Expand-Contract Pattern**: Add new → migrate data → remove old\n2. **Backward Compatible**: Old app version must work with new schema\n3. **Forward Compatible**: New app version must work with old schema\n4. **Non-Blocking**: Avoid long-running locks on tables\n```\n\n### Adding a Column\n\n```sql\n-- Phase 1: Add nullable column (backward compatible)\nALTER TABLE data.users ADD COLUMN phone text;\n\n-- Phase 2: Deploy new app that writes to both old and new\n-- Phase 3: Backfill data\nUPDATE data.users SET phone = extract_phone_from_profile(profile) WHERE phone IS NULL;\n\n-- Phase 4: Add NOT NULL constraint (after all data migrated)\nALTER TABLE data.users ALTER COLUMN phone SET NOT NULL;\n```\n\n### Renaming a Column\n\n```sql\n-- ❌ Bad: Breaks old app immediately\nALTER TABLE data.users RENAME COLUMN name TO full_name;\n\n-- ✅ Good: Expand-contract pattern\n\n-- Phase 1: Add new column\nALTER TABLE data.users ADD COLUMN full_name text;\n\n-- Phase 2: Create trigger to sync both columns\nCREATE FUNCTION private.sync_user_names()\nRETURNS trigger LANGUAGE plpgsql AS $\nBEGIN\n IF TG_OP = 'INSERT' OR NEW.name IS DISTINCT FROM OLD.name THEN\n NEW.full_name := NEW.name;\n END IF;\n IF TG_OP = 'INSERT' OR NEW.full_name IS DISTINCT FROM OLD.full_name THEN\n NEW.name := NEW.full_name;\n END IF;\n RETURN NEW;\nEND;\n$;\n\nCREATE TRIGGER users_sync_names_trg\n BEFORE INSERT OR UPDATE ON data.users\n FOR EACH ROW EXECUTE FUNCTION private.sync_user_names();\n\n-- Phase 3: Backfill\nUPDATE data.users SET full_name = name WHERE full_name IS NULL;\n\n-- Phase 4: Deploy new app using full_name\n-- Phase 5: Remove old column and trigger (after old app retired)\nDROP TRIGGER users_sync_names_trg ON data.users;\nDROP FUNCTION private.sync_user_names();\nALTER TABLE data.users DROP COLUMN name;\n```\n\n### Changing Column Type\n\n```sql\n-- ❌ Bad: Locks table, may fail with data\nALTER TABLE data.orders ALTER COLUMN status TYPE integer USING status::integer;\n\n-- ✅ Good: Add new column, migrate, swap\n\n-- Phase 1: Add new column\nALTER TABLE data.orders ADD COLUMN status_new integer;\n\n-- Phase 2: Sync trigger\nCREATE FUNCTION private.sync_order_status()\nRETURNS trigger LANGUAGE plpgsql AS $\nBEGIN\n NEW.status_new := CASE NEW.status\n WHEN 'pending' THEN 1\n WHEN 'processing' THEN 2\n WHEN 'completed' THEN 3\n ELSE 0\n END;\n RETURN NEW;\nEND;\n$;\n\nCREATE TRIGGER orders_sync_status_trg\n BEFORE INSERT OR UPDATE OF status ON data.orders\n FOR EACH ROW EXECUTE FUNCTION private.sync_order_status();\n\n-- Phase 3: Backfill\nUPDATE data.orders SET status_new = CASE status\n WHEN 'pending' THEN 1 WHEN 'processing' THEN 2 WHEN 'completed' THEN 3 ELSE 0\nEND WHERE status_new IS NULL;\n\n-- Phase 4: Update API functions to use status_new\n-- Phase 5: Drop old column\n```\n\n### Adding NOT NULL Constraint\n\n```sql\n-- ❌ Bad: Full table scan with exclusive lock\nALTER TABLE data.large_table ALTER COLUMN email SET NOT NULL;\n\n-- ✅ Good: NOT VALID then validate separately\n\n-- Phase 1: Add constraint without validation (no lock)\nALTER TABLE data.large_table\n ADD CONSTRAINT large_table_email_not_null\n CHECK (email IS NOT NULL) NOT VALID;\n\n-- Phase 2: Ensure no NULLs in new data (app/trigger enforces)\n\n-- Phase 3: Validate existing data (ShareUpdateExclusiveLock, allows reads/writes)\nALTER TABLE data.large_table VALIDATE CONSTRAINT large_table_email_not_null;\n\n-- Phase 4: Optionally convert to true NOT NULL\n-- (Requires PG18 or accept the CHECK constraint)\n```\n\n### Creating Index Without Downtime\n\n```sql\n-- ❌ Bad: Blocks writes\nCREATE INDEX orders_email_idx ON data.orders(email);\n\n-- ✅ Good: CONCURRENTLY (allows writes during build)\nCREATE INDEX CONCURRENTLY orders_email_idx ON data.orders(email);\n\n-- Note: CONCURRENTLY cannot be in a transaction\n-- If it fails, the index may be left invalid:\nSELECT indexrelid::regclass, indisvalid\nFROM pg_index\nWHERE NOT indisvalid;\n\n-- Rebuild invalid index\nREINDEX INDEX CONCURRENTLY orders_email_idx;\n```\n\n### Dropping a Column\n\n```sql\n-- ❌ Bad: Immediate drop may break old app\nALTER TABLE data.users DROP COLUMN legacy_field;\n\n-- ✅ Good: Mark unused, then drop later\n\n-- Phase 1: Stop writing to column (app change)\n-- Phase 2: Deprecate in comments\nCOMMENT ON COLUMN data.users.legacy_field IS 'DEPRECATED: Do not use. Will be removed.';\n\n-- Phase 3: (Optional) Rename to make usage obvious\nALTER TABLE data.users RENAME COLUMN legacy_field TO _deprecated_legacy_field;\n\n-- Phase 4: Drop after verification period\nALTER TABLE data.users DROP COLUMN _deprecated_legacy_field;\n```\n\n### Migration Checklist for Zero-Downtime\n\n```markdown\n## Pre-Migration\n- [ ] Schema change is backward compatible\n- [ ] Schema change is forward compatible\n- [ ] No exclusive locks on large tables\n- [ ] Tested in staging with production-like data\n- [ ] Rollback plan documented\n\n## During Migration\n- [ ] Monitor query performance\n- [ ] Monitor lock waits\n- [ ] Monitor replication lag\n\n## Post-Migration\n- [ ] Verify data integrity\n- [ ] Clean up temporary objects\n- [ ] Update documentation\n```\n\n## Complete Implementation\n\n### Full Migration System Setup Script\n\n```sql\n-- ============================================================\n-- DATABASE MIGRATION SYSTEM\n-- ============================================================\n-- Run this script to initialize the migration system\n-- ============================================================\n\nBEGIN;\n\n-- Create migration schema\nCREATE SCHEMA IF NOT EXISTS app_migration;\n\n-- Changelog table\nCREATE TABLE IF NOT EXISTS app_migration.changelog (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n version text NOT NULL,\n description text NOT NULL,\n type text NOT NULL DEFAULT 'versioned',\n filename text NOT NULL,\n checksum text NOT NULL,\n execution_time_ms integer,\n executed_at timestamptz NOT NULL DEFAULT now(),\n executed_by text NOT NULL DEFAULT current_user,\n success boolean NOT NULL DEFAULT true,\n \n CONSTRAINT changelog_type_check \n CHECK (type IN ('versioned', 'repeatable', 'baseline'))\n);\n\nCREATE UNIQUE INDEX IF NOT EXISTS changelog_version_key \n ON app_migration.changelog(version) \n WHERE type = 'versioned' AND success = true;\n\n-- Lock configuration\nCREATE TABLE IF NOT EXISTS app_migration.lock_config (\n id integer PRIMARY KEY DEFAULT 1,\n lock_id bigint NOT NULL DEFAULT 123456789,\n CONSTRAINT lock_config_single_row CHECK (id = 1)\n);\n\nINSERT INTO app_migration.lock_config DEFAULT VALUES\nON CONFLICT DO NOTHING;\n\n-- Rollback scripts storage\nCREATE TABLE IF NOT EXISTS app_migration.rollback_scripts (\n version text PRIMARY KEY,\n rollback_sql text NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Rollback history\nCREATE TABLE IF NOT EXISTS app_migration.rollback_log (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n changelog_id bigint NOT NULL,\n version text NOT NULL,\n rollback_sql text,\n rolled_back_at timestamptz NOT NULL DEFAULT now(),\n rolled_back_by text NOT NULL DEFAULT current_user,\n success boolean NOT NULL DEFAULT true\n);\n\n-- [Include all functions and procedures from above sections]\n\nCOMMIT;\n\n-- Verify installation\nSELECT * FROM app_migration.info();\n```\n\n### Usage Example\n\n```sql\n-- 1. Acquire lock before running migrations\nSELECT app_migration.acquire_lock();\n\n-- 2. Run a versioned migration (simple helper)\nCALL app_migration.run_versioned(\n in_version := '001',\n in_description := 'Create users table',\n in_sql := $mig$\n CREATE TABLE data.users (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n email text NOT NULL UNIQUE,\n created_at timestamptz NOT NULL DEFAULT now()\n );\n $mig$,\n in_rollback_sql := 'DROP TABLE data.users;'\n);\n\n-- 3. Run repeatable migration\nCALL app_migration.run_repeatable(\n in_filename := 'R__views.sql',\n in_description := 'Application views',\n in_sql := $mig$\n DROP VIEW IF EXISTS api.v_active_users CASCADE;\n CREATE VIEW api.v_active_users AS\n SELECT * FROM data.users WHERE is_active = true;\n $mig$\n);\n\n-- 4. Check migration status\nSELECT * FROM app_migration.info();\nSELECT * FROM app_migration.get_history(10);\n\n-- 5. Release lock\nSELECT app_migration.release_lock();\n\n-- To rollback if needed:\n-- SELECT app_migration.acquire_lock();\n-- CALL app_migration.rollback('001');\n-- SELECT app_migration.release_lock();\n```\n\n> **Note**: The `run_versioned()` and `run_repeatable()` helpers are provided by `002_migration_runner_helpers.sql`. The core `app_migration.execute()` procedure is available from `001_install_migration_system.sql`.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":31592,"content_sha256":"200660f2834756385344cca88b2494f5df8c62943f17a0afad743219ab839751"},{"filename":"references/monitoring-observability.md","content":"# Monitoring and Observability Patterns\n\nThis document covers pg_stat_statements, slow query logging, custom metrics, health checks, and alerting patterns for PostgreSQL.\n\n## Table of Contents\n\n1. [pg_stat_statements](#pg_stat_statements)\n2. [Slow Query Logging](#slow-query-logging)\n3. [Connection Monitoring](#connection-monitoring)\n4. [Table and Index Statistics](#table-and-index-statistics)\n5. [Lock Monitoring](#lock-monitoring)\n6. [Replication Monitoring](#replication-monitoring)\n7. [Custom Metrics](#custom-metrics)\n8. [Health Check Endpoints](#health-check-endpoints)\n9. [Alerting Patterns](#alerting-patterns)\n\n## pg_stat_statements\n\n### Setup\n\n```sql\n-- Enable extension (requires postgresql.conf change and restart)\n-- shared_preload_libraries = 'pg_stat_statements'\n\nCREATE EXTENSION IF NOT EXISTS pg_stat_statements;\n\n-- Configuration options (postgresql.conf)\n-- pg_stat_statements.max = 10000\n-- pg_stat_statements.track = all\n-- pg_stat_statements.track_utility = on\n-- pg_stat_statements.track_planning = on\n```\n\n### Query Analysis\n\n```sql\n-- Top queries by total time\nSELECT \n round(total_exec_time::numeric, 2) AS total_time_ms,\n calls,\n round(mean_exec_time::numeric, 2) AS avg_time_ms,\n round((100 * total_exec_time / sum(total_exec_time) OVER ())::numeric, 2) AS pct_total,\n rows,\n query\nFROM pg_stat_statements\nORDER BY total_exec_time DESC\nLIMIT 20;\n\n-- Top queries by call count\nSELECT \n calls,\n round(total_exec_time::numeric, 2) AS total_time_ms,\n round(mean_exec_time::numeric, 2) AS avg_time_ms,\n rows,\n query\nFROM pg_stat_statements\nORDER BY calls DESC\nLIMIT 20;\n\n-- Slowest queries on average\nSELECT \n round(mean_exec_time::numeric, 2) AS avg_time_ms,\n round(stddev_exec_time::numeric, 2) AS stddev_ms,\n calls,\n query\nFROM pg_stat_statements\nWHERE calls >= 10 -- Filter out rarely executed queries\nORDER BY mean_exec_time DESC\nLIMIT 20;\n```\n\n### Query Performance View\n\n```sql\nCREATE OR REPLACE VIEW api.v_query_performance AS\nSELECT \n queryid,\n calls,\n round(total_exec_time::numeric, 2) AS total_time_ms,\n round(mean_exec_time::numeric, 2) AS avg_time_ms,\n round(min_exec_time::numeric, 2) AS min_time_ms,\n round(max_exec_time::numeric, 2) AS max_time_ms,\n round(stddev_exec_time::numeric, 2) AS stddev_ms,\n rows,\n round((100.0 * shared_blks_hit / nullif(shared_blks_hit + shared_blks_read, 0))::numeric, 2) AS cache_hit_pct,\n left(query, 200) AS query_preview\nFROM pg_stat_statements\nWHERE dbid = (SELECT oid FROM pg_database WHERE datname = current_database())\nORDER BY total_exec_time DESC;\n\n-- Reset statistics periodically\nSELECT pg_stat_statements_reset();\n```\n\n### Tracking Query Changes Over Time\n\n```sql\n-- Store snapshots for trend analysis\nCREATE TABLE app_audit.query_stats_snapshots (\n snapshot_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n snapshot_at timestamptz NOT NULL DEFAULT now(),\n queryid bigint NOT NULL,\n calls bigint,\n total_time_ms numeric,\n avg_time_ms numeric,\n rows bigint\n);\n\n-- Procedure to capture snapshot\nCREATE OR REPLACE PROCEDURE app_audit.capture_query_stats()\nLANGUAGE plpgsql\nAS $\nBEGIN\n INSERT INTO app_audit.query_stats_snapshots \n (snapshot_at, queryid, calls, total_time_ms, avg_time_ms, rows)\n SELECT \n now(),\n queryid,\n calls,\n round(total_exec_time::numeric, 2),\n round(mean_exec_time::numeric, 2),\n rows\n FROM pg_stat_statements\n WHERE calls >= 100; -- Only track frequently used queries\nEND;\n$;\n\n-- Schedule with pg_cron (hourly)\nSELECT cron.schedule('query-stats-snapshot', '0 * * * *', \n 'CALL app_audit.capture_query_stats()');\n```\n\n## Slow Query Logging\n\n### PostgreSQL Configuration\n\n```ini\n# postgresql.conf\n\n# Log queries slower than 1 second\nlog_min_duration_statement = 1000\n\n# Log all statements (for debugging - use sparingly)\n# log_statement = 'all'\n\n# Log detailed execution stats\nlog_duration = on\nlog_lock_waits = on\nlog_temp_files = 0 # Log all temp file usage\n\n# Log format for parsing\nlog_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h '\n\n# Log to CSV for analysis\nlog_destination = 'csvlog'\nlogging_collector = on\nlog_directory = 'log'\nlog_filename = 'postgresql-%Y-%m-%d.log'\n```\n\n### Slow Query Analysis Table\n\n```sql\n-- Table to store analyzed slow queries\nCREATE TABLE app_audit.slow_queries (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n logged_at timestamptz NOT NULL,\n duration_ms numeric NOT NULL,\n query text NOT NULL,\n user_name text,\n database_name text,\n application text,\n client_addr inet,\n query_hash text GENERATED ALWAYS AS (md5(query)) STORED,\n analyzed boolean NOT NULL DEFAULT false,\n notes text\n);\n\nCREATE INDEX slow_queries_logged_idx ON app_audit.slow_queries(logged_at);\nCREATE INDEX slow_queries_duration_idx ON app_audit.slow_queries(duration_ms DESC);\nCREATE INDEX slow_queries_hash_idx ON app_audit.slow_queries(query_hash);\n\n-- Function to import from CSV log\nCREATE OR REPLACE PROCEDURE app_audit.import_slow_queries(\n in_log_file text\n)\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Create temp table for CSV import\n CREATE TEMP TABLE temp_log (\n log_time timestamptz,\n user_name text,\n database_name text,\n process_id int,\n connection_from text,\n session_id text,\n session_line_num bigint,\n command_tag text,\n session_start_time timestamptz,\n virtual_transaction_id text,\n transaction_id bigint,\n error_severity text,\n sql_state_code text,\n message text,\n detail text,\n hint text,\n internal_query text,\n internal_query_pos int,\n context text,\n query text,\n query_pos int,\n location text,\n application_name text\n );\n \n EXECUTE format('COPY temp_log FROM %L WITH CSV', in_log_file);\n \n INSERT INTO app_audit.slow_queries (logged_at, duration_ms, query, user_name, database_name, application)\n SELECT \n log_time,\n (regexp_match(message, 'duration: ([0-9.]+) ms'))[1]::numeric,\n query,\n user_name,\n database_name,\n application_name\n FROM temp_log\n WHERE message LIKE 'duration:%'\n AND query IS NOT NULL;\n \n DROP TABLE temp_log;\nEND;\n$;\n```\n\n## Connection Monitoring\n\n### Connection Statistics View\n\n```sql\nCREATE OR REPLACE VIEW api.v_connection_stats AS\nWITH connection_counts AS (\n SELECT \n usename,\n application_name,\n client_addr,\n state,\n wait_event_type,\n COUNT(*) AS count\n FROM pg_stat_activity\n WHERE datname = current_database()\n GROUP BY usename, application_name, client_addr, state, wait_event_type\n)\nSELECT \n usename,\n application_name,\n state,\n wait_event_type,\n SUM(count) AS connections,\n ROUND(100.0 * SUM(count) / (SELECT COUNT(*) FROM pg_stat_activity WHERE datname = current_database()), 2) AS pct\nFROM connection_counts\nGROUP BY usename, application_name, state, wait_event_type\nORDER BY connections DESC;\n\n-- Connection pool status\nCREATE OR REPLACE FUNCTION api.get_connection_pool_status()\nRETURNS TABLE (\n total_connections int,\n active_connections int,\n idle_connections int,\n idle_in_transaction int,\n waiting_connections int,\n max_connections int,\n connection_utilization_pct numeric\n)\nLANGUAGE sql\nSTABLE\nAS $\n SELECT \n (SELECT COUNT(*)::int FROM pg_stat_activity WHERE datname = current_database()),\n (SELECT COUNT(*)::int FROM pg_stat_activity WHERE datname = current_database() AND state = 'active'),\n (SELECT COUNT(*)::int FROM pg_stat_activity WHERE datname = current_database() AND state = 'idle'),\n (SELECT COUNT(*)::int FROM pg_stat_activity WHERE datname = current_database() AND state LIKE 'idle in transaction%'),\n (SELECT COUNT(*)::int FROM pg_stat_activity WHERE datname = current_database() AND wait_event_type = 'Lock'),\n current_setting('max_connections')::int,\n ROUND(100.0 * (SELECT COUNT(*) FROM pg_stat_activity) / current_setting('max_connections')::int, 2);\n$;\n```\n\n### Long-Running Query Detection\n\n```sql\nCREATE OR REPLACE FUNCTION api.get_long_running_queries(\n in_threshold interval DEFAULT interval '5 minutes'\n)\nRETURNS TABLE (\n pid int,\n duration interval,\n state text,\n query text,\n waiting boolean,\n username text,\n application text,\n client_addr inet\n)\nLANGUAGE sql\nSTABLE\nAS $\n SELECT \n pid,\n now() - query_start AS duration,\n state,\n left(query, 500),\n wait_event IS NOT NULL,\n usename,\n application_name,\n client_addr\n FROM pg_stat_activity\n WHERE datname = current_database()\n AND state != 'idle'\n AND query_start \u003c now() - in_threshold\n AND pid != pg_backend_pid()\n ORDER BY query_start;\n$;\n```\n\n## Table and Index Statistics\n\n### Table Health View\n\n```sql\nCREATE OR REPLACE VIEW api.v_table_health AS\nSELECT \n schemaname,\n relname AS table_name,\n n_live_tup AS live_rows,\n n_dead_tup AS dead_rows,\n ROUND(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_row_pct,\n last_vacuum,\n last_autovacuum,\n last_analyze,\n last_autoanalyze,\n vacuum_count,\n autovacuum_count,\n pg_size_pretty(pg_total_relation_size(schemaname || '.' || relname)) AS total_size\nFROM pg_stat_user_tables\nWHERE schemaname = 'data'\nORDER BY n_dead_tup DESC;\n\n-- Index usage statistics\nCREATE OR REPLACE VIEW api.v_index_usage AS\nSELECT \n schemaname,\n tablename,\n indexname,\n idx_scan AS index_scans,\n idx_tup_read AS tuples_read,\n idx_tup_fetch AS tuples_fetched,\n pg_size_pretty(pg_relation_size(indexrelid)) AS index_size,\n CASE \n WHEN idx_scan = 0 THEN 'UNUSED'\n WHEN idx_scan \u003c 100 THEN 'LOW USAGE'\n ELSE 'ACTIVE'\n END AS usage_status\nFROM pg_stat_user_indexes\nWHERE schemaname = 'data'\nORDER BY idx_scan;\n```\n\n### Bloat Detection\n\n```sql\n-- Estimate table bloat\nCREATE OR REPLACE VIEW api.v_table_bloat AS\nWITH constants AS (\n SELECT current_setting('block_size')::numeric AS bs\n),\ntable_stats AS (\n SELECT \n schemaname,\n tablename,\n reltuples::bigint AS row_count,\n relpages::bigint AS page_count,\n pg_relation_size(schemaname || '.' || tablename) AS actual_size\n FROM pg_stat_user_tables\n JOIN pg_class ON relname = tablename\n WHERE schemaname = 'data'\n)\nSELECT \n schemaname,\n tablename,\n row_count,\n pg_size_pretty(actual_size) AS actual_size,\n pg_size_pretty((page_count * (SELECT bs FROM constants))::bigint) AS page_size,\n ROUND(100.0 * (actual_size - page_count * (SELECT bs FROM constants)) / NULLIF(actual_size, 0), 2) AS bloat_pct\nFROM table_stats\nWHERE actual_size > 1024 * 1024 -- Only tables > 1MB\nORDER BY actual_size DESC;\n```\n\n## Lock Monitoring\n\n### Current Locks View\n\n```sql\nCREATE OR REPLACE VIEW api.v_current_locks AS\nSELECT \n l.pid,\n l.mode,\n l.granted,\n l.locktype,\n CASE l.locktype\n WHEN 'relation' THEN c.relname\n WHEN 'virtualxid' THEN l.virtualxid::text\n WHEN 'transactionid' THEN l.transactionid::text\n ELSE l.locktype\n END AS locked_object,\n a.usename,\n a.application_name,\n a.state,\n a.query_start,\n now() - a.query_start AS duration,\n left(a.query, 200) AS query\nFROM pg_locks l\nJOIN pg_stat_activity a ON l.pid = a.pid\nLEFT JOIN pg_class c ON l.relation = c.oid\nWHERE l.pid != pg_backend_pid()\nORDER BY a.query_start;\n\n-- Lock wait chains (who's blocking whom)\nCREATE OR REPLACE FUNCTION api.get_blocking_queries()\nRETURNS TABLE (\n blocked_pid int,\n blocked_user text,\n blocked_query text,\n blocked_duration interval,\n blocking_pid int,\n blocking_user text,\n blocking_query text\n)\nLANGUAGE sql\nSTABLE\nAS $\n SELECT \n blocked.pid AS blocked_pid,\n blocked.usename AS blocked_user,\n left(blocked.query, 200) AS blocked_query,\n now() - blocked.query_start AS blocked_duration,\n blocking.pid AS blocking_pid,\n blocking.usename AS blocking_user,\n left(blocking.query, 200) AS blocking_query\n FROM pg_locks blocked_locks\n JOIN pg_stat_activity blocked ON blocked_locks.pid = blocked.pid\n JOIN pg_locks blocking_locks ON (\n blocking_locks.locktype = blocked_locks.locktype AND\n blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database AND\n blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation AND\n blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page AND\n blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple AND\n blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid AND\n blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid AND\n blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid AND\n blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid AND\n blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid AND\n blocking_locks.pid != blocked_locks.pid\n )\n JOIN pg_stat_activity blocking ON blocking_locks.pid = blocking.pid\n WHERE NOT blocked_locks.granted\n AND blocking_locks.granted;\n$;\n```\n\n## Replication Monitoring\n\n### Replication Status\n\n```sql\n-- Check replication status (on primary)\nCREATE OR REPLACE VIEW api.v_replication_status AS\nSELECT \n client_addr,\n usename,\n application_name,\n state,\n sync_state,\n sent_lsn,\n write_lsn,\n flush_lsn,\n replay_lsn,\n pg_size_pretty(pg_wal_lsn_diff(sent_lsn, replay_lsn)) AS replication_lag,\n write_lag,\n flush_lag,\n replay_lag\nFROM pg_stat_replication;\n\n-- Replication lag in seconds\nCREATE OR REPLACE FUNCTION api.get_replication_lag_seconds()\nRETURNS TABLE (\n replica text,\n lag_bytes bigint,\n lag_seconds numeric\n)\nLANGUAGE sql\nSTABLE\nAS $\n SELECT \n client_addr::text || ' (' || application_name || ')',\n pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn),\n EXTRACT(EPOCH FROM replay_lag)\n FROM pg_stat_replication;\n$;\n```\n\n## Custom Metrics\n\n### Metrics Collection Tables\n\n```sql\nCREATE TABLE app_audit.metrics (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n metric_name text NOT NULL,\n metric_value numeric NOT NULL,\n labels jsonb DEFAULT '{}',\n collected_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE INDEX metrics_name_time_idx ON app_audit.metrics(metric_name, collected_at);\n\n-- Partition by time for efficiency\n-- (See partitioning section for implementation)\n```\n\n### Metrics Collection Procedure\n\n```sql\nCREATE OR REPLACE PROCEDURE app_audit.collect_metrics()\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_metric_time timestamptz := now();\nBEGIN\n -- Connection metrics\n INSERT INTO app_audit.metrics (metric_name, metric_value, labels, collected_at)\n SELECT \n 'pg_connections_total',\n COUNT(*),\n jsonb_build_object('state', state),\n l_metric_time\n FROM pg_stat_activity\n WHERE datname = current_database()\n GROUP BY state;\n \n -- Transaction metrics\n INSERT INTO app_audit.metrics (metric_name, metric_value, labels, collected_at)\n SELECT \n 'pg_stat_database_xact_commit',\n xact_commit,\n jsonb_build_object('database', datname),\n l_metric_time\n FROM pg_stat_database\n WHERE datname = current_database();\n \n INSERT INTO app_audit.metrics (metric_name, metric_value, labels, collected_at)\n SELECT \n 'pg_stat_database_xact_rollback',\n xact_rollback,\n jsonb_build_object('database', datname),\n l_metric_time\n FROM pg_stat_database\n WHERE datname = current_database();\n \n -- Cache hit ratio\n INSERT INTO app_audit.metrics (metric_name, metric_value, collected_at)\n SELECT \n 'pg_cache_hit_ratio',\n round(100.0 * sum(heap_blks_hit) / nullif(sum(heap_blks_hit) + sum(heap_blks_read), 0), 2),\n l_metric_time\n FROM pg_statio_user_tables;\n \n -- Table sizes\n INSERT INTO app_audit.metrics (metric_name, metric_value, labels, collected_at)\n SELECT \n 'pg_table_size_bytes',\n pg_total_relation_size(schemaname || '.' || tablename),\n jsonb_build_object('schema', schemaname, 'table', tablename),\n l_metric_time\n FROM pg_tables\n WHERE schemaname = 'data';\n \n -- Dead tuple ratio\n INSERT INTO app_audit.metrics (metric_name, metric_value, labels, collected_at)\n SELECT \n 'pg_dead_tuple_ratio',\n round(100.0 * n_dead_tup / nullif(n_live_tup + n_dead_tup, 0), 2),\n jsonb_build_object('table', relname),\n l_metric_time\n FROM pg_stat_user_tables\n WHERE schemaname = 'data'\n AND n_live_tup > 0;\nEND;\n$;\n\n-- Schedule collection (every minute)\nSELECT cron.schedule('collect-metrics', '* * * * *', \n 'CALL app_audit.collect_metrics()');\n```\n\n### Prometheus-Compatible Metrics Endpoint\n\n```sql\n-- Function that returns metrics in Prometheus format\nCREATE OR REPLACE FUNCTION api.metrics_prometheus()\nRETURNS text\nLANGUAGE plpgsql\nSTABLE\nAS $\nDECLARE\n l_output text := '';\n l_record RECORD;\nBEGIN\n -- Database connections\n l_output := l_output || '# HELP pg_connections_total Number of database connections' || E'\\n';\n l_output := l_output || '# TYPE pg_connections_total gauge' || E'\\n';\n FOR l_record IN\n SELECT state, COUNT(*) AS count\n FROM pg_stat_activity\n WHERE datname = current_database()\n GROUP BY state\n LOOP\n l_output := l_output || format('pg_connections_total{state=\"%s\"} %s', l_record.state, l_record.count) || E'\\n';\n END LOOP;\n \n -- Cache hit ratio\n l_output := l_output || E'\\n# HELP pg_cache_hit_ratio Cache hit ratio' || E'\\n';\n l_output := l_output || '# TYPE pg_cache_hit_ratio gauge' || E'\\n';\n l_output := l_output || format('pg_cache_hit_ratio %s', (\n SELECT round(100.0 * sum(heap_blks_hit) / nullif(sum(heap_blks_hit) + sum(heap_blks_read), 0), 2)\n FROM pg_statio_user_tables\n )) || E'\\n';\n \n -- Transaction rate\n l_output := l_output || E'\\n# HELP pg_xact_commit_total Transactions committed' || E'\\n';\n l_output := l_output || '# TYPE pg_xact_commit_total counter' || E'\\n';\n l_output := l_output || format('pg_xact_commit_total %s', (\n SELECT xact_commit FROM pg_stat_database WHERE datname = current_database()\n )) || E'\\n';\n \n RETURN l_output;\nEND;\n$;\n```\n\n## Health Check Endpoints\n\n### Comprehensive Health Check\n\n```sql\nCREATE OR REPLACE FUNCTION api.healthcheck()\nRETURNS jsonb\nLANGUAGE plpgsql\nSTABLE\nAS $\nDECLARE\n l_result jsonb;\n l_start_time timestamptz := clock_timestamp();\n l_checks jsonb := '{}';\n l_healthy boolean := true;\nBEGIN\n -- Database connectivity\n l_checks := l_checks || jsonb_build_object(\n 'database', jsonb_build_object(\n 'status', 'ok',\n 'latency_ms', round(extract(milliseconds from clock_timestamp() - l_start_time)::numeric, 2)\n )\n );\n \n -- Connection pool\n DECLARE\n l_conn_count int;\n l_max_conn int;\n l_util_pct numeric;\n BEGIN\n SELECT COUNT(*) INTO l_conn_count FROM pg_stat_activity WHERE datname = current_database();\n l_max_conn := current_setting('max_connections')::int;\n l_util_pct := round(100.0 * l_conn_count / l_max_conn, 2);\n \n l_checks := l_checks || jsonb_build_object(\n 'connections', jsonb_build_object(\n 'status', CASE WHEN l_util_pct \u003c 80 THEN 'ok' WHEN l_util_pct \u003c 95 THEN 'warning' ELSE 'critical' END,\n 'current', l_conn_count,\n 'max', l_max_conn,\n 'utilization_pct', l_util_pct\n )\n );\n \n IF l_util_pct >= 95 THEN l_healthy := false; END IF;\n END;\n \n -- Replication lag (if replica)\n DECLARE\n l_lag_bytes bigint;\n BEGIN\n SELECT pg_wal_lsn_diff(pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn())\n INTO l_lag_bytes;\n \n IF l_lag_bytes IS NOT NULL THEN\n l_checks := l_checks || jsonb_build_object(\n 'replication', jsonb_build_object(\n 'status', CASE WHEN l_lag_bytes \u003c 1048576 THEN 'ok' WHEN l_lag_bytes \u003c 104857600 THEN 'warning' ELSE 'critical' END,\n 'lag_bytes', l_lag_bytes,\n 'lag_mb', round(l_lag_bytes / 1048576.0, 2)\n )\n );\n \n IF l_lag_bytes >= 104857600 THEN l_healthy := false; END IF;\n END IF;\n EXCEPTION\n WHEN OTHERS THEN NULL; -- Not a replica\n END;\n \n -- Long-running queries\n DECLARE\n l_long_queries int;\n BEGIN\n SELECT COUNT(*) INTO l_long_queries\n FROM pg_stat_activity\n WHERE state = 'active'\n AND query_start \u003c now() - interval '5 minutes'\n AND pid != pg_backend_pid();\n \n l_checks := l_checks || jsonb_build_object(\n 'long_queries', jsonb_build_object(\n 'status', CASE WHEN l_long_queries = 0 THEN 'ok' WHEN l_long_queries \u003c 5 THEN 'warning' ELSE 'critical' END,\n 'count', l_long_queries\n )\n );\n \n IF l_long_queries >= 5 THEN l_healthy := false; END IF;\n END;\n \n -- Dead tuple ratio (vacuum needed?)\n DECLARE\n l_max_dead_ratio numeric;\n BEGIN\n SELECT MAX(round(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2))\n INTO l_max_dead_ratio\n FROM pg_stat_user_tables\n WHERE schemaname = 'data' AND n_live_tup > 1000;\n \n l_checks := l_checks || jsonb_build_object(\n 'vacuum', jsonb_build_object(\n 'status', CASE WHEN COALESCE(l_max_dead_ratio, 0) \u003c 10 THEN 'ok' WHEN l_max_dead_ratio \u003c 30 THEN 'warning' ELSE 'critical' END,\n 'max_dead_tuple_ratio', COALESCE(l_max_dead_ratio, 0)\n )\n );\n END;\n \n l_result := jsonb_build_object(\n 'status', CASE WHEN l_healthy THEN 'healthy' ELSE 'unhealthy' END,\n 'timestamp', now(),\n 'response_time_ms', round(extract(milliseconds from clock_timestamp() - l_start_time)::numeric, 2),\n 'checks', l_checks\n );\n \n RETURN l_result;\nEND;\n$;\n\n-- Simple liveness check\nCREATE OR REPLACE FUNCTION api.liveness()\nRETURNS text\nLANGUAGE sql\nAS $\n SELECT 'ok';\n$;\n\n-- Readiness check (can accept traffic?)\nCREATE OR REPLACE FUNCTION api.readiness()\nRETURNS jsonb\nLANGUAGE plpgsql\nSTABLE\nAS $\nDECLARE\n l_ready boolean := true;\n l_issues text[] := '{}';\nBEGIN\n -- Check connection pool\n IF (SELECT COUNT(*) FROM pg_stat_activity) > current_setting('max_connections')::int * 0.95 THEN\n l_ready := false;\n l_issues := array_append(l_issues, 'Connection pool near capacity');\n END IF;\n \n -- Check for lock contention\n IF (SELECT COUNT(*) FROM pg_locks WHERE NOT granted) > 10 THEN\n l_ready := false;\n l_issues := array_append(l_issues, 'High lock contention');\n END IF;\n \n RETURN jsonb_build_object(\n 'ready', l_ready,\n 'issues', to_jsonb(l_issues)\n );\nEND;\n$;\n```\n\n## Alerting Patterns\n\n### Alert Rules Table\n\n```sql\nCREATE TABLE app_audit.alert_rules (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n name text NOT NULL UNIQUE,\n description text,\n query text NOT NULL,\n threshold numeric NOT NULL,\n comparison text NOT NULL CHECK (comparison IN ('>', '\u003c', '>=', '\u003c=', '=', '!=')),\n severity text NOT NULL CHECK (severity IN ('info', 'warning', 'critical')),\n is_enabled boolean NOT NULL DEFAULT true,\n cooldown_minutes int NOT NULL DEFAULT 15,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE TABLE app_audit.alert_history (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n rule_id bigint REFERENCES app_audit.alert_rules(id),\n fired_at timestamptz NOT NULL DEFAULT now(),\n current_value numeric,\n message text,\n acknowledged_at timestamptz,\n acknowledged_by text\n);\n\n-- Default alert rules\nINSERT INTO app_audit.alert_rules (name, description, query, threshold, comparison, severity)\nVALUES \n ('high_connections', 'Connection pool utilization > 80%',\n 'SELECT 100.0 * COUNT(*) / current_setting(''max_connections'')::int FROM pg_stat_activity',\n 80, '>', 'warning'),\n \n ('critical_connections', 'Connection pool utilization > 95%',\n 'SELECT 100.0 * COUNT(*) / current_setting(''max_connections'')::int FROM pg_stat_activity',\n 95, '>', 'critical'),\n \n ('long_running_queries', 'Queries running > 5 minutes',\n 'SELECT COUNT(*) FROM pg_stat_activity WHERE state = ''active'' AND query_start \u003c now() - interval ''5 minutes''',\n 0, '>', 'warning'),\n \n ('replication_lag_mb', 'Replication lag > 100MB',\n 'SELECT COALESCE(MAX(pg_wal_lsn_diff(sent_lsn, replay_lsn)) / 1048576.0, 0) FROM pg_stat_replication',\n 100, '>', 'critical'),\n \n ('dead_tuple_ratio', 'Tables with > 20% dead tuples',\n 'SELECT COUNT(*) FROM pg_stat_user_tables WHERE n_dead_tup > n_live_tup * 0.2 AND n_live_tup > 1000',\n 0, '>', 'warning');\n```\n\n### Alert Checking Procedure\n\n```sql\nCREATE OR REPLACE PROCEDURE app_audit.check_alerts()\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_rule RECORD;\n l_value numeric;\n l_should_fire boolean;\n l_last_fired timestamptz;\nBEGIN\n FOR l_rule IN \n SELECT * FROM app_audit.alert_rules WHERE is_enabled\n LOOP\n -- Execute the check query\n EXECUTE l_rule.query INTO l_value;\n \n -- Evaluate condition\n l_should_fire := CASE l_rule.comparison\n WHEN '>' THEN l_value > l_rule.threshold\n WHEN '\u003c' THEN l_value \u003c l_rule.threshold\n WHEN '>=' THEN l_value >= l_rule.threshold\n WHEN '\u003c=' THEN l_value \u003c= l_rule.threshold\n WHEN '=' THEN l_value = l_rule.threshold\n WHEN '!=' THEN l_value != l_rule.threshold\n END;\n \n IF l_should_fire THEN\n -- Check cooldown\n SELECT MAX(fired_at) INTO l_last_fired\n FROM app_audit.alert_history\n WHERE rule_id = l_rule.id;\n \n IF l_last_fired IS NULL OR l_last_fired \u003c now() - (l_rule.cooldown_minutes || ' minutes')::interval THEN\n -- Fire alert\n INSERT INTO app_audit.alert_history (rule_id, current_value, message)\n VALUES (\n l_rule.id,\n l_value,\n format('[%s] %s: current value %s %s threshold %s',\n l_rule.severity, l_rule.name, l_value, l_rule.comparison, l_rule.threshold)\n );\n \n -- Here you could also call a notification function\n -- PERFORM private.send_alert_notification(l_rule, l_value);\n \n RAISE NOTICE 'Alert fired: %', l_rule.name;\n END IF;\n END IF;\n END LOOP;\nEND;\n$;\n\n-- Schedule alert checks (every minute)\nSELECT cron.schedule('check-alerts', '* * * * *', \n 'CALL app_audit.check_alerts()');\n```\n\n### View Active Alerts\n\n```sql\nCREATE OR REPLACE VIEW api.v_active_alerts AS\nSELECT \n r.name AS alert_name,\n r.severity,\n h.fired_at,\n h.current_value,\n h.message,\n h.acknowledged_at IS NOT NULL AS is_acknowledged,\n h.acknowledged_by\nFROM app_audit.alert_history h\nJOIN app_audit.alert_rules r ON r.id = h.rule_id\nWHERE h.fired_at > now() - interval '24 hours'\nORDER BY h.fired_at DESC;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":27906,"content_sha256":"0b3633ebd0ddae672ac961ec01b77859b8c0c96729c7977986d8758e5f04b253"},{"filename":"references/oracle-migration-guide.md","content":"# Oracle to PostgreSQL Migration Guide\n\nA reference guide for developers familiar with Oracle PL/SQL transitioning to PostgreSQL PL/pgSQL. This covers syntax differences, equivalent patterns, and common gotchas.\n\n## Table of Contents\n\n1. [Data Type Mapping](#data-type-mapping)\n2. [SQL Syntax Differences](#sql-syntax-differences)\n3. [PL/SQL to PL/pgSQL](#plsql-to-plpgsql)\n4. [Packages to Schemas](#packages-to-schemas)\n5. [Sequences and Identity](#sequences-and-identity)\n6. [Date and Time Handling](#date-and-time-handling)\n7. [String Functions](#string-functions)\n8. [NULL Handling](#null-handling)\n9. [Transactions and Locking](#transactions-and-locking)\n10. [Error Handling](#error-handling)\n11. [Common Gotchas](#common-gotchas)\n\n## Data Type Mapping\n\n| Oracle | PostgreSQL | Notes |\n|--------|------------|-------|\n| `VARCHAR2(n)` | `text` or `varchar(n)` | Prefer `text` - no performance penalty |\n| `NVARCHAR2(n)` | `text` | PostgreSQL is UTF-8 native |\n| `CHAR(n)` | `char(n)` or `text` | Avoid `char` - use `text` |\n| `NUMBER` | `numeric` | Arbitrary precision |\n| `NUMBER(p)` | `numeric(p)` or `bigint` | Use `bigint` for integers |\n| `NUMBER(p,s)` | `numeric(p,s)` | Exact match |\n| `INTEGER` | `integer` | Same |\n| `FLOAT` | `double precision` | IEEE 754 |\n| `BINARY_FLOAT` | `real` | 32-bit float |\n| `BINARY_DOUBLE` | `double precision` | 64-bit float |\n| `DATE` | `timestamp` | Oracle DATE includes time! |\n| `TIMESTAMP` | `timestamp` | Same |\n| `TIMESTAMP WITH TIME ZONE` | `timestamptz` | Same |\n| `INTERVAL YEAR TO MONTH` | `interval` | PostgreSQL interval is more flexible |\n| `INTERVAL DAY TO SECOND` | `interval` | Same |\n| `CLOB` | `text` | PostgreSQL text is unlimited |\n| `BLOB` | `bytea` | Binary data |\n| `RAW(n)` | `bytea` | Binary data |\n| `LONG` | `text` | Deprecated in Oracle anyway |\n| `LONG RAW` | `bytea` | Deprecated |\n| `BOOLEAN` | `boolean` | Oracle doesn't have native boolean! |\n| `ROWID` | `ctid` | Different semantics - avoid |\n| `XMLType` | `xml` | Native XML support |\n| `JSON` | `jsonb` | Use `jsonb` for efficiency |\n| `SYS_REFCURSOR` | `refcursor` | Similar concept |\n\n### Important Notes\n\n```sql\n-- Oracle DATE includes time (common mistake!)\n-- Oracle:\nSELECT SYSDATE FROM dual; -- Returns date AND time\n\n-- PostgreSQL:\nSELECT now(); -- timestamp with time zone\nSELECT CURRENT_DATE; -- date only\nSELECT CURRENT_TIMESTAMP; -- timestamp with time zone\n```\n\n## SQL Syntax Differences\n\n### SELECT Differences\n\n```sql\n-- Oracle: DUAL table for expressions\nSELECT 1 + 1 FROM dual;\n\n-- PostgreSQL: No FROM needed\nSELECT 1 + 1;\n\n-- Oracle: ROWNUM for limiting\nSELECT * FROM customers WHERE ROWNUM \u003c= 10;\n\n-- PostgreSQL: LIMIT/OFFSET\nSELECT * FROM customers LIMIT 10;\nSELECT * FROM customers LIMIT 10 OFFSET 20;\n\n-- Oracle: Hierarchical queries with CONNECT BY\nSELECT * FROM employees\nSTART WITH manager_id IS NULL\nCONNECT BY PRIOR employee_id = manager_id;\n\n-- PostgreSQL: Recursive CTE\nWITH RECURSIVE emp_tree AS (\n SELECT employee_id, name, manager_id, 1 AS level\n FROM employees\n WHERE manager_id IS NULL\n \n UNION ALL\n \n SELECT e.employee_id, e.name, e.manager_id, et.level + 1\n FROM employees e\n JOIN emp_tree et ON e.manager_id = et.employee_id\n)\nSELECT * FROM emp_tree;\n```\n\n### Outer Joins\n\n```sql\n-- Oracle: Old-style (+) syntax (avoid)\nSELECT * FROM orders o, customers c\nWHERE o.customer_id = c.id(+);\n\n-- PostgreSQL: ANSI join (use this in Oracle too!)\nSELECT * FROM orders o\nLEFT JOIN customers c ON c.id = o.customer_id;\n```\n\n### Merge Statement\n\n```sql\n-- Oracle: MERGE\nMERGE INTO products p\nUSING new_products np ON (p.sku = np.sku)\nWHEN MATCHED THEN UPDATE SET p.price = np.price\nWHEN NOT MATCHED THEN INSERT (sku, price) VALUES (np.sku, np.price);\n\n-- PostgreSQL: INSERT ON CONFLICT\nINSERT INTO products (sku, price)\nSELECT sku, price FROM new_products\nON CONFLICT (sku) DO UPDATE SET price = EXCLUDED.price;\n```\n\n### Sequences\n\n```sql\n-- Oracle\nCREATE SEQUENCE order_seq START WITH 1 INCREMENT BY 1;\nSELECT order_seq.NEXTVAL FROM dual;\nSELECT order_seq.CURRVAL FROM dual;\n\n-- PostgreSQL\nCREATE SEQUENCE order_seq START WITH 1 INCREMENT BY 1;\nSELECT nextval('order_seq');\nSELECT currval('order_seq');\n\n-- PostgreSQL: Better - use IDENTITY\nCREATE TABLE orders (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY\n);\n```\n\n## PL/SQL to PL/pgSQL\n\n### Basic Syntax\n\n```sql\n-- Oracle PL/SQL\nCREATE OR REPLACE PROCEDURE update_salary(\n p_employee_id IN NUMBER,\n p_new_salary IN NUMBER\n) AS\n v_old_salary NUMBER;\nBEGIN\n SELECT salary INTO v_old_salary\n FROM employees\n WHERE employee_id = p_employee_id;\n \n UPDATE employees\n SET salary = p_new_salary\n WHERE employee_id = p_employee_id;\n \n DBMS_OUTPUT.PUT_LINE('Updated from ' || v_old_salary || ' to ' || p_new_salary);\nEXCEPTION\n WHEN NO_DATA_FOUND THEN\n RAISE_APPLICATION_ERROR(-20001, 'Employee not found');\nEND;\n/\n\n-- PostgreSQL PL/pgSQL\nCREATE OR REPLACE PROCEDURE api.update_salary(\n in_employee_id bigint,\n in_new_salary numeric\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_old_salary numeric;\nBEGIN\n SELECT salary INTO l_old_salary\n FROM data.employees\n WHERE employee_id = in_employee_id;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Employee not found: %', in_employee_id\n USING ERRCODE = 'P0001';\n END IF;\n \n UPDATE data.employees\n SET salary = in_new_salary\n WHERE employee_id = in_employee_id;\n \n RAISE NOTICE 'Updated from % to %', l_old_salary, in_new_salary;\nEND;\n$;\n```\n\n### Key Differences\n\n| Oracle PL/SQL | PostgreSQL PL/pgSQL |\n|---------------|---------------------|\n| `CREATE OR REPLACE PROCEDURE name AS` | `CREATE OR REPLACE PROCEDURE name() LANGUAGE plpgsql AS $` |\n| `p_param IN NUMBER` | `in_param numeric` (no IN keyword needed) |\n| `p_param OUT NUMBER` | Use `INOUT` or return value |\n| `p_param IN OUT NUMBER` | `INOUT io_param numeric` |\n| `v_variable NUMBER;` | `l_variable numeric;` (in DECLARE) |\n| `v_variable := value;` | `l_variable := value;` |\n| `DBMS_OUTPUT.PUT_LINE()` | `RAISE NOTICE '%', message;` |\n| `RAISE_APPLICATION_ERROR()` | `RAISE EXCEPTION '' USING ERRCODE = '';` |\n| `NO_DATA_FOUND` exception | Check `FOUND` variable or `NOT FOUND` |\n| `SQL%ROWCOUNT` | `GET DIAGNOSTICS var = ROW_COUNT;` |\n| `/` to execute | `;` to execute |\n\n### Functions\n\n```sql\n-- Oracle\nCREATE OR REPLACE FUNCTION get_customer_name(p_id IN NUMBER)\nRETURN VARCHAR2 AS\n v_name VARCHAR2(100);\nBEGIN\n SELECT name INTO v_name FROM customers WHERE id = p_id;\n RETURN v_name;\nEXCEPTION\n WHEN NO_DATA_FOUND THEN\n RETURN NULL;\nEND;\n/\n\n-- PostgreSQL\nCREATE OR REPLACE FUNCTION api.get_customer_name(in_id bigint)\nRETURNS text\nLANGUAGE plpgsql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_name text;\nBEGIN\n SELECT name INTO l_name \n FROM data.customers \n WHERE id = in_id;\n \n RETURN l_name; -- Returns NULL if not found\nEND;\n$;\n```\n\n### Cursors\n\n```sql\n-- Oracle\nDECLARE\n CURSOR c_orders IS\n SELECT order_id, total FROM orders WHERE status = 'PENDING';\n v_order c_orders%ROWTYPE;\nBEGIN\n OPEN c_orders;\n LOOP\n FETCH c_orders INTO v_order;\n EXIT WHEN c_orders%NOTFOUND;\n -- Process v_order\n END LOOP;\n CLOSE c_orders;\nEND;\n\n-- PostgreSQL (using FOR loop - preferred)\nDO $\nDECLARE\n r_order RECORD;\nBEGIN\n FOR r_order IN \n SELECT order_id, total FROM orders WHERE status = 'pending'\n LOOP\n -- Process r_order\n RAISE NOTICE 'Order: %, Total: %', r_order.order_id, r_order.total;\n END LOOP;\nEND;\n$;\n\n-- PostgreSQL (explicit cursor if needed)\nDO $\nDECLARE\n c_orders CURSOR FOR\n SELECT order_id, total FROM orders WHERE status = 'pending';\n r_order RECORD;\nBEGIN\n OPEN c_orders;\n LOOP\n FETCH c_orders INTO r_order;\n EXIT WHEN NOT FOUND;\n -- Process r_order\n END LOOP;\n CLOSE c_orders;\nEND;\n$;\n```\n\n### Bulk Operations\n\n```sql\n-- Oracle: BULK COLLECT and FORALL\nDECLARE\n TYPE t_ids IS TABLE OF NUMBER;\n TYPE t_names IS TABLE OF VARCHAR2(100);\n v_ids t_ids;\n v_names t_names;\nBEGIN\n SELECT id, name BULK COLLECT INTO v_ids, v_names\n FROM customers WHERE status = 'ACTIVE';\n \n FORALL i IN v_ids.FIRST..v_ids.LAST\n UPDATE orders SET customer_name = v_names(i)\n WHERE customer_id = v_ids(i);\nEND;\n\n-- PostgreSQL: Use set-based operations (no FORALL needed)\nUPDATE orders o\nSET customer_name = c.name\nFROM customers c\nWHERE c.id = o.customer_id\n AND c.status = 'active';\n\n-- PostgreSQL: If you need arrays\nDO $\nDECLARE\n t_ids uuid[];\nBEGIN\n SELECT array_agg(id) INTO t_ids\n FROM customers WHERE status = 'active';\n \n UPDATE orders SET processed = true\n WHERE customer_id = ANY(t_ids);\nEND;\n$;\n```\n\n## Packages to Schemas\n\nOracle packages provide namespacing, public/private separation, and state. PostgreSQL schemas provide similar namespacing.\n\n```sql\n-- Oracle Package Specification\nCREATE OR REPLACE PACKAGE customers_pkg AS\n -- Public procedures/functions\n FUNCTION get_customer(p_id NUMBER) RETURN customers%ROWTYPE;\n PROCEDURE insert_customer(p_email VARCHAR2, p_name VARCHAR2);\n PROCEDURE update_status(p_id NUMBER, p_status VARCHAR2);\n \n -- Package variable (session state)\n g_default_status VARCHAR2(20) := 'ACTIVE';\nEND customers_pkg;\n/\n\n-- Oracle Package Body\nCREATE OR REPLACE PACKAGE BODY customers_pkg AS\n -- Private function\n FUNCTION validate_email(p_email VARCHAR2) RETURN BOOLEAN IS\n BEGIN\n RETURN REGEXP_LIKE(p_email, '^[^@]+@[^@]+\\.[^@]+

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

);\n END;\n \n -- Public implementations\n FUNCTION get_customer(p_id NUMBER) RETURN customers%ROWTYPE IS\n v_customer customers%ROWTYPE;\n BEGIN\n SELECT * INTO v_customer FROM customers WHERE id = p_id;\n RETURN v_customer;\n END;\n \n -- ... other implementations\nEND customers_pkg;\n/\n```\n\n```sql\n-- PostgreSQL: Use schemas for namespacing\n\n-- Public functions in api schema\nCREATE OR REPLACE FUNCTION api.get_customer(in_id uuid)\nRETURNS TABLE (id uuid, email text, name text, status text)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name, status\n FROM data.customers\n WHERE id = in_id;\n$;\n\nCREATE OR REPLACE PROCEDURE api.insert_customer(\n in_email text,\n in_name text,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n IF NOT private.validate_email(in_email) THEN\n RAISE EXCEPTION 'Invalid email format';\n END IF;\n \n INSERT INTO data.customers (email, name, status)\n VALUES (lower(in_email), in_name, private.get_default_status())\n RETURNING id INTO io_id;\nEND;\n$;\n\n-- Private functions in private schema\nCREATE OR REPLACE FUNCTION private.validate_email(in_email text)\nRETURNS boolean\nLANGUAGE sql\nIMMUTABLE\nAS $\n SELECT in_email ~ '^[^@]+@[^@]+\\.[^@]+

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

;\n$;\n\n-- Package variables become session variables or config functions\nCREATE OR REPLACE FUNCTION private.get_default_status()\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nAS $\n SELECT 'active'::text;\n$;\n```\n\n### Package Variable Alternatives\n\n```sql\n-- Oracle package variable\n-- customers_pkg.g_current_user_id\n\n-- PostgreSQL: Session variable\nSET myapp.current_user_id = 'user-uuid';\nSELECT current_setting('myapp.current_user_id');\n\n-- PostgreSQL: Function wrapper\nCREATE OR REPLACE FUNCTION private.get_current_user_id()\nRETURNS uuid\nLANGUAGE sql\nSTABLE\nAS $\n SELECT NULLIF(current_setting('myapp.current_user_id', true), '')::uuid;\n$;\n```\n\n### Oracle-Package Style: Sub-Schemas + File-per-Module\n\nFor teams porting Oracle PL/SQL codebases, the single flat `api` schema loses the *package* mental model — the collapsible, namespaced unit that holds every routine for one domain. The pattern below restores it by giving each Oracle-package equivalent its own sub-schema (`api_customers`, `api_orders`, …) and its own source file. Combined, they deliver the closest thing PostgreSQL has to Oracle packages: per-domain folders in your IDE, per-package grants, and one file per \"package body.\"\n\n#### When to Use This Pattern\n\nThis is the recommended layout for **Oracle-migration projects**. The package mental model is so deeply ingrained in PL/SQL teams that the familiarity benefit can justify sub-schemas earlier than the canonical 50+ tables / 200+ functions threshold documented in [coding-standards-trivadis.md → Option 2](coding-standards-trivadis.md#option-2-sub-schema-pattern-for-large-applications). That said:\n\n- **Smaller projects** (under ~20 tables) without an Oracle background should still start with the canonical single `api` schema described in [schema-architecture.md](schema-architecture.md). Sub-schemas add real overhead in grants, search_path, and migration runners.\n- **Larger projects** with clean domain boundaries — and especially anything porting from Oracle — benefit from sub-schemas regardless of exact table count. The bigger the codebase, the more the IDE-tree grouping pays off.\n\n#### Package Map\n\nA 12-table online-store example, mapped to 10 packages. Tightly-coupled child tables fold into their parent's package — same way Oracle would bundle `orders` and `order_items` operations in one `orders_pkg`.\n\n| Package schema | Tables it owns | Notes |\n|---|---|---|\n| `api_customers` | `data.customers` | |\n| `api_addresses` | `data.addresses` | |\n| `api_categories` | `data.categories` | |\n| `api_products` | `data.products` | |\n| `api_inventory` | `data.inventory` | |\n| `api_carts` | `data.carts`, `data.cart_items` | child folds in |\n| `api_orders` | `data.orders`, `data.order_items` | child folds in |\n| `api_payments` | `data.payments` | |\n| `api_shipments` | `data.shipments` | |\n| `api_reviews` | `data.reviews` | |\n\n10 packages cover 12 tables. Note that **tables stay in `data`** — only the routines get sub-schema-ized. Do not create `data_orders` or `data_products`; that fragments foreign keys and complicates joins.\n\n#### Schema Creation\n\n```sql\n-- Core storage and internals\nCREATE SCHEMA data;\nCREATE SCHEMA private;\n\n-- One \"package\" per API domain\nCREATE SCHEMA api_customers;\nCREATE SCHEMA api_addresses;\nCREATE SCHEMA api_categories;\nCREATE SCHEMA api_products;\nCREATE SCHEMA api_inventory;\nCREATE SCHEMA api_carts;\nCREATE SCHEMA api_orders;\nCREATE SCHEMA api_payments;\nCREATE SCHEMA api_shipments;\nCREATE SCHEMA api_reviews;\n\n-- Lock down the default namespace\nREVOKE ALL ON SCHEMA public FROM PUBLIC;\n```\n\nSchema names use the **plural** convention to mirror table naming (`data.orders` → `api_orders`), consistent with [coding-standards-trivadis.md](coding-standards-trivadis.md#database-object-naming).\n\n#### Directory Layout (File-per-Module)\n\n```\nsql/\n├── 000_schemas.sql -- CREATE SCHEMA statements above\n│\n├── data/ -- Tables, indexes, constraints\n│ ├── 010_customers.sql\n│ ├── 011_addresses.sql\n│ ├── 012_categories.sql\n│ ├── 013_products.sql\n│ ├── 014_inventory.sql\n│ ├── 015_carts.sql -- carts + cart_items together\n│ ├── 016_orders.sql -- orders + order_items together\n│ ├── 017_payments.sql\n│ ├── 018_shipments.sql\n│ └── 019_reviews.sql\n│\n├── private/ -- Helpers + trigger functions\n│ ├── 100_set_updated_at.sql\n│ ├── 101_log_audit.sql\n│ └── 110_triggers.sql\n│\n├── api/ -- One file per package (Oracle \"package body\")\n│ ├── 200_customers.sql -- api_customers.*\n│ ├── 201_addresses.sql -- api_addresses.*\n│ ├── 202_categories.sql -- api_categories.*\n│ ├── 203_products.sql -- api_products.*\n│ ├── 204_inventory.sql -- api_inventory.*\n│ ├── 205_carts.sql -- api_carts.*\n│ ├── 206_orders.sql -- api_orders.*\n│ ├── 207_payments.sql -- api_payments.*\n│ ├── 208_shipments.sql -- api_shipments.*\n│ └── 209_reviews.sql -- api_reviews.*\n│\n└── grants/\n └── 900_grants.sql -- GRANT USAGE / EXECUTE per package\n```\n\n**Numeric prefixes** give deterministic load order in any `psql -f` loop or migration runner, and IDEs render them in sequence. The `200_*` band reserves room for 99 packages before colliding with the next band.\n\n#### Package Routine Inventory\n\nEach package contains the same kinds of routines you'd find in an Oracle package body. Names get shorter because the package context lives in the schema name.\n\n##### `api_customers` (file: `sql/api/200_customers.sql`)\n\n- `api_customers.get_by_id(in_id uuid)`\n- `api_customers.get_by_email(in_email text)`\n- `api_customers.select_verified()`\n- `api_customers.insert(in_email, in_full_name, INOUT io_id)`\n- `api_customers.update(in_id, in_full_name)`\n- `api_customers.delete(in_id)`\n\n##### `api_addresses` (file: `201_addresses.sql`)\n\n- `api_addresses.get_by_id(in_id)`\n- `api_addresses.select_by_customer(in_customer_id)`\n- `api_addresses.insert(...)`\n- `api_addresses.update(...)`\n- `api_addresses.delete(in_id)`\n\n##### `api_categories` (file: `202_categories.sql`)\n\n- `api_categories.get_by_id(in_id)`\n- `api_categories.select_all()`\n- `api_categories.select_by_parent(in_parent_id)`\n- `api_categories.insert(...)`\n- `api_categories.update(...)`\n- `api_categories.delete(in_id)`\n\n##### `api_products` (file: `203_products.sql`)\n\n- `api_products.get_by_id(in_id)`\n- `api_products.get_by_sku(in_sku)`\n- `api_products.select_by_category(in_category_id)`\n- `api_products.select_by_category_and_active(in_category_id, in_is_active)`\n- `api_products.calculate_rating(in_id)`\n- `api_products.insert(...)`\n- `api_products.update(...)`\n- `api_products.upsert(...)`\n- `api_products.delete(in_id)`\n\n##### `api_inventory` (file: `204_inventory.sql`)\n\n- `api_inventory.get_by_product(in_product_id)`\n- `api_inventory.select_by_warehouse(in_warehouse_id)`\n- `api_inventory.calculate_available_stock(in_product_id)`\n- `api_inventory.update_quantity(in_id, in_qty)`\n\n##### `api_carts` (file: `205_carts.sql`) — covers carts + cart_items\n\n- `api_carts.get_by_id(in_id)`\n- `api_carts.get_by_customer(in_customer_id)`\n- `api_carts.insert(in_customer_id, INOUT io_id)`\n- `api_carts.delete(in_id)`\n- `api_carts.item_select_by_cart(in_cart_id)`\n- `api_carts.item_insert(...)`\n- `api_carts.item_upsert(...)`\n- `api_carts.item_update_quantity(in_id, in_qty)`\n- `api_carts.item_delete(in_id)`\n\n##### `api_orders` (file: `206_orders.sql`) — covers orders + order_items\n\n- `api_orders.get_by_id(in_id)`\n- `api_orders.select_by_customer(in_customer_id)`\n- `api_orders.select_by_status_and_date(in_status, in_from, in_to)`\n- `api_orders.calculate_total(in_id)`\n- `api_orders.insert(in_customer_id, INOUT io_id)`\n- `api_orders.update_status(in_id, in_new_status)`\n- `api_orders.delete(in_id)`\n- `api_orders.item_select_by_order(in_order_id)`\n- `api_orders.item_insert(...)`\n- `api_orders.item_delete(in_id)`\n\n##### `api_payments` (file: `207_payments.sql`)\n\n- `api_payments.get_by_id(in_id)`\n- `api_payments.select_by_order(in_order_id)`\n- `api_payments.validate(in_id)`\n- `api_payments.insert(...)`\n- `api_payments.update_status(in_id, in_new_status)`\n\n##### `api_shipments` (file: `208_shipments.sql`)\n\n- `api_shipments.get_by_id(in_id)`\n- `api_shipments.select_by_order(in_order_id)`\n- `api_shipments.insert(...)`\n- `api_shipments.update_status(in_id, in_new_status)`\n\n##### `api_reviews` (file: `209_reviews.sql`)\n\n- `api_reviews.get_by_id(in_id)`\n- `api_reviews.select_by_product(in_product_id)`\n- `api_reviews.select_by_customer(in_customer_id)`\n- `api_reviews.insert(...)`\n- `api_reviews.update(...)`\n- `api_reviews.delete(in_id)`\n\n#### Sample Package File: `sql/api/206_orders.sql`\n\nThe full \"package body\" for `api_orders`. Everything for orders + order_items lives in one file:\n\n```sql\n-- =============================================================================\n-- api_orders package\n-- Owns: data.orders, data.order_items\n-- Depends on: data (read/write), private (helpers, triggers)\n-- =============================================================================\n\n-- ─── READS ───────────────────────────────────────────────────────────────────\n\nCREATE OR REPLACE FUNCTION api_orders.get_by_id(in_order_id uuid)\nRETURNS TABLE (id uuid, customer_id uuid, status text, total_amount numeric, placed_at timestamptz)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, customer_id, status, total_amount, placed_at\n FROM data.orders\n WHERE id = in_order_id;\n$;\n\nCREATE OR REPLACE FUNCTION api_orders.select_by_customer(\n in_customer_id uuid,\n in_limit integer DEFAULT 100,\n in_offset integer DEFAULT 0\n)\nRETURNS TABLE (id uuid, status text, total_amount numeric, placed_at timestamptz)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, status, total_amount, placed_at\n FROM data.orders\n WHERE customer_id = in_customer_id\n ORDER BY created_at DESC\n LIMIT in_limit OFFSET in_offset;\n$;\n\nCREATE OR REPLACE FUNCTION api_orders.select_by_status_and_date(\n in_status text,\n in_start_date timestamptz,\n in_end_date timestamptz\n)\nRETURNS TABLE (id uuid, customer_id uuid, total_amount numeric, placed_at timestamptz)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, customer_id, total_amount, placed_at\n FROM data.orders\n WHERE status = in_status\n AND placed_at >= in_start_date\n AND placed_at \u003c in_end_date\n ORDER BY placed_at DESC;\n$;\n\nCREATE OR REPLACE FUNCTION api_orders.calculate_total(in_order_id uuid)\nRETURNS numeric\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT COALESCE(SUM(quantity * unit_price), 0)\n FROM data.order_items\n WHERE order_id = in_order_id;\n$;\n\n-- ─── WRITES ──────────────────────────────────────────────────────────────────\n\nCREATE OR REPLACE PROCEDURE api_orders.insert(\n in_customer_id uuid,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.orders (customer_id)\n VALUES (in_customer_id)\n RETURNING id INTO io_id;\nEND;\n$;\n\nCREATE OR REPLACE PROCEDURE api_orders.update_status(\n in_order_id uuid,\n in_new_status text\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n UPDATE data.orders\n SET status = in_new_status,\n updated_at = now()\n WHERE id = in_order_id;\nEND;\n$;\n\nCREATE OR REPLACE PROCEDURE api_orders.delete(in_order_id uuid)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n DELETE FROM data.orders WHERE id = in_order_id;\nEND;\n$;\n\n-- ─── CHILD: order_items (lives in the same package) ─────────────────────────\n\nCREATE OR REPLACE FUNCTION api_orders.item_select_by_order(in_order_id uuid)\nRETURNS TABLE (id uuid, product_id uuid, quantity integer, unit_price numeric)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, product_id, quantity, unit_price\n FROM data.order_items\n WHERE order_id = in_order_id;\n$;\n\nCREATE OR REPLACE PROCEDURE api_orders.item_insert(\n in_order_id uuid,\n in_product_id uuid,\n in_quantity integer,\n in_unit_price numeric,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.order_items (order_id, product_id, quantity, unit_price)\n VALUES (in_order_id, in_product_id, in_quantity, in_unit_price)\n RETURNING id INTO io_id;\nEND;\n$;\n\nCREATE OR REPLACE PROCEDURE api_orders.item_delete(in_id uuid)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n DELETE FROM data.order_items WHERE id = in_id;\nEND;\n$;\n\n-- ─── PACKAGE METADATA ────────────────────────────────────────────────────────\n\nCOMMENT ON SCHEMA api_orders IS\n 'Orders package — orders and order_items. Owns data.orders, data.order_items.';\n```\n\nEvery routine uses `SECURITY DEFINER SET search_path = data, private, pg_temp` and parameters are prefixed `in_` / `io_` per Trivadis convention. This file *is* the package body — readable top to bottom, no jumping between files.\n\n#### Per-Package Grants\n\nThe biggest operational win of sub-schemas: **least-privilege grants happen at schema granularity**, not function-by-function.\n\n```sql\n-- sql/grants/900_grants.sql\n\n-- App role gets every package\nGRANT USAGE ON SCHEMA api_customers, api_addresses, api_categories,\n api_products, api_inventory, api_carts,\n api_orders, api_payments, api_shipments,\n api_reviews\n TO app_role;\n\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA api_customers TO app_role;\nGRANT EXECUTE ON ALL PROCEDURES IN SCHEMA api_customers TO app_role;\n-- ... repeat per package, or wrap in a DO block\n\n-- Fine-grained: shipping microservice gets only what it needs\nGRANT USAGE ON SCHEMA api_shipments, api_orders TO shipping_svc;\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA api_shipments TO shipping_svc;\nGRANT EXECUTE ON ALL PROCEDURES IN SCHEMA api_shipments TO shipping_svc;\nGRANT EXECUTE ON FUNCTION api_orders.get_by_id(uuid) TO shipping_svc;\n```\n\nNo function-by-function whitelisting. New routines added to a package are automatically covered (combined with `ALTER DEFAULT PRIVILEGES`).\n\n#### Optional: search_path for Callers\n\nIf you want callers to use short-form references like `SELECT orders.get_by_id(...)` rather than fully-qualified names, set the search_path on the role:\n\n```sql\nALTER ROLE app_role SET search_path = api_customers, api_addresses, api_categories,\n api_products, api_inventory, api_carts,\n api_orders, api_payments, api_shipments,\n api_reviews, pg_catalog;\n```\n\nIn practice, fully-qualified `api_orders.get_by_id(...)` is more self-documenting at the call site and avoids silent shadowing if two packages happen to expose a routine with the same name. Configuring search_path is mostly useful for ad-hoc `psql` sessions.\n\n#### IDE Tree — The Payoff\n\nIn DataGrip / DBeaver / pgAdmin, the database browser renders each package as a collapsible folder of 5–10 routines:\n\n```\npostgres\n├── data ← tables only\n│ ├── customers\n│ ├── addresses\n│ ├── orders\n│ ├── order_items\n│ └── ...\n├── private ← helpers, triggers\n│\n├── api_carts ← collapse/expand like an Oracle package\n│ ├── Functions\n│ │ ├── get_by_customer\n│ │ ├── get_by_id\n│ │ └── item_select_by_cart\n│ └── Procedures\n│ ├── delete\n│ ├── insert\n│ ├── item_delete\n│ ├── item_insert\n│ ├── item_update_quantity\n│ └── item_upsert\n│\n├── api_orders ← each package is a tidy, small folder\n│ ├── Functions\n│ │ ├── calculate_total\n│ │ ├── get_by_id\n│ │ ├── item_select_by_order\n│ │ ├── select_by_customer\n│ │ └── select_by_status_and_date\n│ └── Procedures\n│ ├── delete\n│ ├── insert\n│ ├── item_delete\n│ ├── item_insert\n│ └── update_status\n│\n├── api_products\n│ └── ...\n└── ...\n```\n\nCompared to a flat 60+ routine `api` schema, navigation becomes **pick package → pick routine** instead of search/filter. That is the Oracle package mental model, restored.\n\n#### psql Introspection\n\n```\n\\dn api_* -- list all packages\n\\df api_orders.* -- list all routines in a package\n\\df+ api_orders.get_by_id -- full signature + COMMENT\n```\n\nThis is the equivalent of Oracle's `DESC customers_pkg` — fast, terminal-native package introspection.\n\n#### Design Rules That Keep Boundaries Clean\n\nThese five rules keep package boundaries from eroding over time:\n\n1. **Packages do not call each other's `api_*` functions directly.** If `api_orders` needs product data, it reads `data.products` directly. Cross-package API-to-API calls create hidden coupling and blur permission boundaries.\n2. **Shared helpers live in `private`**, not in an `api_common` package. If multiple packages need the same logic, factor it into a `private` function and call it from each.\n3. **One file = one package = one `api_*` schema.** Never split a package across multiple files; never mix two packages in one file. The 1:1:1 mapping is the whole point.\n4. **Child tables stay in their parent's package** unless they become independently useful. `order_items` belongs in `api_orders`. Promote a child to its own package only when external systems need direct access to it.\n5. **Tables stay in `data`** — only routines get sub-schema-ized. Do not create `data_orders` or `data_products`; that fragments foreign keys and complicates joins.\n\n#### Cross-References\n\n- [`coding-standards-trivadis.md` → Option 2: Sub-Schema Pattern](coding-standards-trivadis.md#option-2-sub-schema-pattern-for-large-applications) — brief overview and the canonical 50+ table / 200+ function warning.\n- [`schema-architecture.md`](schema-architecture.md) — the canonical single `api` schema pattern, recommended as the default for non-Oracle teams and smaller projects.\n- [`schema-naming.md` → Function & Procedure Naming](schema-naming.md#function--procedure-naming) — the `{action}_{entity}` and `in_`/`io_` parameter conventions used throughout the routines above.\n\n## Sequences and Identity\n\n```sql\n-- Oracle: Create sequence\nCREATE SEQUENCE emp_seq START WITH 1 INCREMENT BY 1;\n\n-- Oracle: Use in INSERT\nINSERT INTO employees (id, name) VALUES (emp_seq.NEXTVAL, 'John');\n\n-- Oracle: Create table with sequence\nCREATE TABLE employees (\n id NUMBER DEFAULT emp_seq.NEXTVAL PRIMARY KEY,\n name VARCHAR2(100)\n);\n\n-- PostgreSQL: IDENTITY columns (preferred)\nCREATE TABLE data.employees (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n name text NOT NULL\n);\n\n-- PostgreSQL: Override identity for data migration\nINSERT INTO data.employees (id, name) \nOVERRIDING SYSTEM VALUE\nVALUES (100, 'Migrated Employee');\n\n-- PostgreSQL: Sequence if needed\nCREATE SEQUENCE data.emp_seq;\nSELECT nextval('data.emp_seq');\n\n-- PostgreSQL: UUID (often better than sequences)\nCREATE TABLE data.employees (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL\n);\n```\n\n## Date and Time Handling\n\n```sql\n-- Oracle\nSELECT SYSDATE FROM dual; -- Current date+time\nSELECT SYSTIMESTAMP FROM dual; -- With timezone\nSELECT TRUNC(SYSDATE) FROM dual; -- Date only\nSELECT ADD_MONTHS(SYSDATE, 3) FROM dual; -- Add months\nSELECT MONTHS_BETWEEN(date1, date2) FROM dual; -- Difference\nSELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD') FROM dual;\nSELECT TO_DATE('2024-01-15', 'YYYY-MM-DD') FROM dual;\n\n-- PostgreSQL\nSELECT now(); -- Current timestamp\nSELECT CURRENT_TIMESTAMP; -- Same\nSELECT CURRENT_DATE; -- Date only\nSELECT now() + interval '3 months'; -- Add months\nSELECT age(date1, date2); -- Difference as interval\nSELECT to_char(now(), 'YYYY-MM-DD');\nSELECT '2024-01-15'::date; -- Or to_date()\nSELECT date_trunc('day', now()); -- Truncate to day\n\n-- Common conversions\n-- Oracle TRUNC(date) -> PostgreSQL date_trunc('day', timestamp)\n-- Oracle ADD_MONTHS(date, n) -> PostgreSQL date + interval 'n months'\n-- Oracle LAST_DAY(date) -> PostgreSQL (date_trunc('month', date) + interval '1 month - 1 day')::date\n```\n\n## String Functions\n\n| Oracle | PostgreSQL | Notes |\n|--------|------------|-------|\n| `\\|\\|` (concat) | `\\|\\|` | Same |\n| `CONCAT(a, b)` | `concat(a, b)` | Same |\n| `LENGTH(str)` | `length(str)` | Same |\n| `SUBSTR(str, start, len)` | `substring(str from start for len)` or `substr()` | `substr()` works same |\n| `INSTR(str, substr)` | `position(substr in str)` or `strpos()` | |\n| `UPPER(str)` | `upper(str)` | Same |\n| `LOWER(str)` | `lower(str)` | Same |\n| `TRIM(str)` | `trim(str)` | Same |\n| `LTRIM(str)` | `ltrim(str)` | Same |\n| `RTRIM(str)` | `rtrim(str)` | Same |\n| `LPAD(str, len, pad)` | `lpad(str, len, pad)` | Same |\n| `RPAD(str, len, pad)` | `rpad(str, len, pad)` | Same |\n| `REPLACE(str, from, to)` | `replace(str, from, to)` | Same |\n| `REGEXP_LIKE(str, pattern)` | `str ~ pattern` | Different syntax |\n| `REGEXP_REPLACE(str, pattern, repl)` | `regexp_replace(str, pattern, repl)` | Similar |\n| `REGEXP_SUBSTR(str, pattern)` | `substring(str from pattern)` | Different |\n| `NVL(val, default)` | `COALESCE(val, default)` | COALESCE is standard SQL |\n| `NVL2(val, if_not_null, if_null)` | `CASE WHEN val IS NOT NULL THEN ... ELSE ... END` | No direct equivalent |\n| `DECODE(val, match1, result1, ...)` | `CASE val WHEN match1 THEN result1 ... END` | Use CASE |\n\n## NULL Handling\n\n```sql\n-- Oracle NVL -> PostgreSQL COALESCE\n-- Oracle\nSELECT NVL(commission, 0) FROM employees;\n\n-- PostgreSQL\nSELECT COALESCE(commission, 0) FROM employees;\n\n-- Oracle NVL2 -> PostgreSQL CASE\n-- Oracle\nSELECT NVL2(commission, 'Has Commission', 'No Commission') FROM employees;\n\n-- PostgreSQL\nSELECT CASE WHEN commission IS NOT NULL \n THEN 'Has Commission' \n ELSE 'No Commission' \n END FROM employees;\n\n-- Oracle DECODE -> PostgreSQL CASE\n-- Oracle\nSELECT DECODE(status, 'A', 'Active', 'I', 'Inactive', 'Unknown') FROM customers;\n\n-- PostgreSQL\nSELECT CASE status \n WHEN 'A' THEN 'Active'\n WHEN 'I' THEN 'Inactive'\n ELSE 'Unknown'\n END FROM customers;\n\n-- Empty string vs NULL\n-- Oracle: empty string equals NULL (usually)\n-- PostgreSQL: empty string is NOT NULL\nSELECT '' IS NULL; -- Oracle: TRUE, PostgreSQL: FALSE\n```\n\n## Transactions and Locking\n\n```sql\n-- Oracle: Implicit transaction start\nUPDATE customers SET status = 'INACTIVE' WHERE id = 1;\nCOMMIT;\n\n-- PostgreSQL: Same (autocommit off by default in psql)\nUPDATE customers SET status = 'inactive' WHERE id = 1;\nCOMMIT;\n\n-- Savepoints (both same)\nSAVEPOINT my_savepoint;\n-- do something\nROLLBACK TO SAVEPOINT my_savepoint;\n\n-- Oracle: SELECT FOR UPDATE WAIT\nSELECT * FROM orders WHERE id = 1 FOR UPDATE WAIT 5;\n\n-- PostgreSQL: No WAIT, use lock_timeout\nSET lock_timeout = '5s';\nSELECT * FROM orders WHERE id = 1 FOR UPDATE;\n\n-- PostgreSQL: SKIP LOCKED (very useful!)\nSELECT * FROM orders WHERE status = 'pending'\nFOR UPDATE SKIP LOCKED\nLIMIT 10;\n\n-- Oracle: Autonomous transactions\n-- PostgreSQL: Use dblink or separate connection\n```\n\n## Error Handling\n\n```sql\n-- Oracle\nBEGIN\n -- do something\nEXCEPTION\n WHEN NO_DATA_FOUND THEN\n -- handle\n WHEN DUP_VAL_ON_INDEX THEN\n -- handle\n WHEN OTHERS THEN\n DBMS_OUTPUT.PUT_LINE('Error: ' || SQLERRM);\n RAISE;\nEND;\n\n-- PostgreSQL\nBEGIN\n -- do something\nEXCEPTION\n WHEN no_data_found THEN\n -- handle (rarely needed - use IF NOT FOUND)\n WHEN unique_violation THEN\n -- handle\n WHEN OTHERS THEN\n RAISE NOTICE 'Error: %', SQLERRM;\n RAISE; -- Re-raise\nEND;\n```\n\n| Oracle Exception | PostgreSQL Exception |\n|------------------|---------------------|\n| `NO_DATA_FOUND` | `no_data_found` (but use `FOUND` variable instead) |\n| `TOO_MANY_ROWS` | `too_many_rows` |\n| `DUP_VAL_ON_INDEX` | `unique_violation` |\n| `VALUE_ERROR` | `data_exception` |\n| `ZERO_DIVIDE` | `division_by_zero` |\n| `INVALID_NUMBER` | `invalid_text_representation` |\n| Custom `-20001` | Custom `ERRCODE` like `'P0001'` |\n\n```sql\n-- Oracle: RAISE_APPLICATION_ERROR\nRAISE_APPLICATION_ERROR(-20001, 'Custom error message');\n\n-- PostgreSQL: RAISE EXCEPTION with ERRCODE\nRAISE EXCEPTION 'Custom error message'\n USING ERRCODE = 'P0001',\n HINT = 'Check your input',\n DETAIL = 'Additional details here';\n```\n\n## Common Gotchas\n\n### 1. Oracle DATE vs PostgreSQL timestamp\n\n```sql\n-- Oracle DATE includes time!\n-- If you're comparing dates, this matters\n\n-- Oracle: This might miss rows from the same day\nSELECT * FROM orders WHERE order_date = DATE '2024-01-15';\n\n-- PostgreSQL: Be explicit\nSELECT * FROM orders \nWHERE order_date >= '2024-01-15'::date \n AND order_date \u003c '2024-01-16'::date;\n\n-- Or use date_trunc\nSELECT * FROM orders \nWHERE date_trunc('day', order_date) = '2024-01-15';\n```\n\n### 2. Case Sensitivity\n\n```sql\n-- Oracle: Identifiers uppercase by default\nSELECT * FROM CUSTOMERS; -- Works\nSELECT * FROM customers; -- Works (same as CUSTOMERS)\nSELECT * FROM \"Customers\"; -- Only works if created with quotes\n\n-- PostgreSQL: Identifiers lowercase by default\nSELECT * FROM CUSTOMERS; -- Becomes: customers\nSELECT * FROM customers; -- Same\nSELECT * FROM \"Customers\"; -- Different! Case-sensitive\n```\n\n### 3. Boolean Type\n\n```sql\n-- Oracle has no native BOOLEAN in SQL (only PL/SQL)\n-- Often uses NUMBER(1) or CHAR(1)\nSELECT * FROM users WHERE is_active = 1;\nSELECT * FROM users WHERE is_active = 'Y';\n\n-- PostgreSQL has native boolean\nSELECT * FROM users WHERE is_active = true;\nSELECT * FROM users WHERE is_active; -- Shorthand\nSELECT * FROM users WHERE NOT is_active;\n```\n\n### 4. Empty String vs NULL\n\n```sql\n-- Oracle: '' is often treated as NULL\nSELECT * FROM customers WHERE name = ''; -- Might return nothing\nSELECT * FROM customers WHERE name IS NULL; -- Might find ''\n\n-- PostgreSQL: '' is NOT NULL\nSELECT '' IS NULL; -- FALSE\nSELECT * FROM customers WHERE name = ''; -- Finds empty strings only\nSELECT * FROM customers WHERE name IS NULL; -- Finds NULLs only\n```\n\n### 5. ROWNUM vs LIMIT\n\n```sql\n-- Oracle ROWNUM is tricky!\n-- This doesn't work as expected:\nSELECT * FROM orders ORDER BY total DESC WHERE ROWNUM \u003c= 10;\n-- ROWNUM is assigned before ORDER BY!\n\n-- Correct Oracle:\nSELECT * FROM (\n SELECT * FROM orders ORDER BY total DESC\n) WHERE ROWNUM \u003c= 10;\n\n-- PostgreSQL is straightforward:\nSELECT * FROM orders ORDER BY total DESC LIMIT 10;\n```\n\n### 6. Automatic Type Coercion\n\n```sql\n-- Oracle is more lenient with types\nSELECT * FROM orders WHERE id = '123'; -- Might work if id is NUMBER\n\n-- PostgreSQL is stricter\nSELECT * FROM orders WHERE id = '123'; -- Error if id is integer\nSELECT * FROM orders WHERE id = 123; -- Correct\nSELECT * FROM orders WHERE id = '123'::integer; -- Explicit cast\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":39372,"content_sha256":"a501a652f5a92146545e398a8c49934b2c7b69584bb748f4d4fe338d0536c15e"},{"filename":"references/partitioning.md","content":"# Partitioning Strategies\n\nThis document covers PostgreSQL declarative partitioning including range, list, and hash partitioning, partition management, and migration strategies.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Partitioning Types](#partitioning-types)\n3. [Range Partitioning](#range-partitioning)\n4. [List Partitioning](#list-partitioning)\n5. [Hash Partitioning](#hash-partitioning)\n6. [Partition Management](#partition-management)\n7. [Indexes on Partitioned Tables](#indexes-on-partitioned-tables)\n8. [Migrating to Partitioned Tables](#migrating-to-partitioned-tables)\n9. [Performance Optimization](#performance-optimization)\n10. [Common Patterns](#common-patterns)\n\n## Overview\n\n### When to Use Partitioning\n\n| Use Case | Benefit | Partitioning Type |\n|----------|---------|-------------------|\n| Time-series data | Fast old data deletion, query pruning | Range |\n| Multi-tenant isolation | Per-tenant management | List |\n| Large table distribution | Parallel operations | Hash |\n| Archival/retention | Drop old partitions | Range |\n| Geographic sharding | Regional queries | List |\n\n### Partitioning Decision Tree\n\n```mermaid\nflowchart TD\n START([Table > 100GB?]) -->|No| SKIP[\"Skip partitioning\u003cbr/>(overhead not worth it)\"]\n START -->|Yes| Q1{Data access pattern?}\n\n Q1 -->|\"Time-based queries\"| RANGE[\"Range Partitioning\u003cbr/>(by date/timestamp)\"]\n Q1 -->|\"Category-based\"| LIST[\"List Partitioning\u003cbr/>(by status, region, tenant)\"]\n Q1 -->|\"Even distribution\"| HASH[\"Hash Partitioning\u003cbr/>(by ID for parallelism)\"]\n Q1 -->|\"Multiple patterns\"| MULTI[\"Sub-partitioning\"]\n\n RANGE --> RETENTION{Need data retention?}\n RETENTION -->|Yes| RANGE_IDEAL[\"✅ Ideal: DROP partition\u003cbr/>instead of DELETE\"]\n RETENTION -->|No| RANGE_OK[\"Good for query pruning\"]\n\n style RANGE_IDEAL fill:#c8e6c9\n style SKIP fill:#fff3e0\n```\n\n### Partitioning Limits\n\n- Maximum 2^14 (16,384) partitions per table\n- Partition key must be included in primary key and unique constraints\n- Foreign keys referencing partitioned tables supported (PostgreSQL 12+)\n- Cross-partition updates require PostgreSQL 11+\n\n## Partitioning Types\n\n### Comparison\n\n| Type | Best For | Key Requirement | Partition Pruning |\n|------|----------|-----------------|-------------------|\n| Range | Time-series, continuous values | Ordered values | Equality, range |\n| List | Categories, regions, tenants | Discrete values | Equality only |\n| Hash | Even distribution | Any hashable type | Equality only |\n\n## Range Partitioning\n\n### Basic Range Partitioning (Time-Based)\n\n```sql\n-- Create partitioned table\nCREATE TABLE data.events (\n id uuid NOT NULL DEFAULT uuidv7(),\n event_type text NOT NULL,\n payload jsonb NOT NULL DEFAULT '{}',\n created_at timestamptz NOT NULL DEFAULT now(),\n\n -- Partition key must be in primary key\n PRIMARY KEY (id, created_at)\n) PARTITION BY RANGE (created_at);\n\n-- Create partitions for each month\nCREATE TABLE data.events_2024_01 PARTITION OF data.events\n FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');\n\nCREATE TABLE data.events_2024_02 PARTITION OF data.events\n FOR VALUES FROM ('2024-02-01') TO ('2024-03-01');\n\nCREATE TABLE data.events_2024_03 PARTITION OF data.events\n FOR VALUES FROM ('2024-03-01') TO ('2024-04-01');\n\n-- Default partition catches any out-of-range values\nCREATE TABLE data.events_default PARTITION OF data.events DEFAULT;\n```\n\n### Range Partition with Tablespaces\n\n```sql\n-- Different tablespaces for hot/cold data\nCREATE TABLE data.events_2024_12 PARTITION OF data.events\n FOR VALUES FROM ('2024-12-01') TO ('2025-01-01')\n TABLESPACE fast_ssd;\n\nCREATE TABLE data.events_2024_01 PARTITION OF data.events\n FOR VALUES FROM ('2024-01-01') TO ('2024-02-01')\n TABLESPACE archive_hdd;\n```\n\n### Numeric Range Partitioning\n\n```sql\n-- Partition by ID range (for sharding-like behavior)\nCREATE TABLE data.users (\n id bigint GENERATED ALWAYS AS IDENTITY,\n email text NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (id)\n) PARTITION BY RANGE (id);\n\nCREATE TABLE data.users_p0 PARTITION OF data.users\n FOR VALUES FROM (1) TO (1000001);\nCREATE TABLE data.users_p1 PARTITION OF data.users\n FOR VALUES FROM (1000001) TO (2000001);\nCREATE TABLE data.users_p2 PARTITION OF data.users\n FOR VALUES FROM (2000001) TO (3000001);\n```\n\n## List Partitioning\n\n### Basic List Partitioning\n\n```sql\n-- Partition by region\nCREATE TABLE data.orders (\n id uuid NOT NULL DEFAULT uuidv7(),\n customer_id uuid NOT NULL,\n region text NOT NULL,\n total numeric(15,2) NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (id, region)\n) PARTITION BY LIST (region);\n\nCREATE TABLE data.orders_us PARTITION OF data.orders\n FOR VALUES IN ('us-east', 'us-west', 'us-central');\n\nCREATE TABLE data.orders_eu PARTITION OF data.orders\n FOR VALUES IN ('eu-west', 'eu-central', 'eu-north');\n\nCREATE TABLE data.orders_apac PARTITION OF data.orders\n FOR VALUES IN ('apac-east', 'apac-south', 'apac-north');\n\nCREATE TABLE data.orders_other PARTITION OF data.orders DEFAULT;\n```\n\n### Multi-Tenant Partitioning\n\n```sql\n-- Each tenant gets its own partition\nCREATE TABLE data.tenant_data (\n id uuid NOT NULL DEFAULT uuidv7(),\n tenant_id uuid NOT NULL,\n data jsonb NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (id, tenant_id)\n) PARTITION BY LIST (tenant_id);\n\n-- Function to create tenant partition\nCREATE FUNCTION private.create_tenant_partition(in_tenant_id uuid)\nRETURNS void\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_partition_name text;\nBEGIN\n l_partition_name := 'tenant_data_' || replace(in_tenant_id::text, '-', '_');\n\n EXECUTE format(\n 'CREATE TABLE data.%I PARTITION OF data.tenant_data FOR VALUES IN (%L)',\n l_partition_name,\n in_tenant_id\n );\n\n RAISE NOTICE 'Created partition: %', l_partition_name;\nEND;\n$;\n\n-- Create partitions for existing tenants\nSELECT private.create_tenant_partition(id) FROM data.tenants;\n```\n\n### Status-Based Partitioning\n\n```sql\n-- Partition by order status (hot/cold separation)\nCREATE TABLE data.orders (\n id uuid NOT NULL DEFAULT uuidv7(),\n status text NOT NULL DEFAULT 'pending',\n total numeric(15,2) NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (id, status),\n CONSTRAINT orders_status_check CHECK (status IN ('pending', 'processing', 'shipped', 'delivered', 'cancelled'))\n) PARTITION BY LIST (status);\n\n-- Active orders (frequently accessed)\nCREATE TABLE data.orders_active PARTITION OF data.orders\n FOR VALUES IN ('pending', 'processing', 'shipped')\n TABLESPACE fast_ssd;\n\n-- Completed orders (rarely accessed)\nCREATE TABLE data.orders_completed PARTITION OF data.orders\n FOR VALUES IN ('delivered', 'cancelled')\n TABLESPACE archive_hdd;\n```\n\n## Hash Partitioning\n\n### Basic Hash Partitioning\n\n```sql\n-- Distribute by customer_id for parallel processing\nCREATE TABLE data.order_items (\n id uuid NOT NULL DEFAULT uuidv7(),\n order_id uuid NOT NULL,\n customer_id uuid NOT NULL,\n product_id uuid NOT NULL,\n quantity integer NOT NULL,\n\n PRIMARY KEY (id, customer_id)\n) PARTITION BY HASH (customer_id);\n\n-- Create 8 partitions (power of 2 recommended)\nCREATE TABLE data.order_items_p0 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 8, REMAINDER 0);\nCREATE TABLE data.order_items_p1 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 8, REMAINDER 1);\nCREATE TABLE data.order_items_p2 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 8, REMAINDER 2);\nCREATE TABLE data.order_items_p3 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 8, REMAINDER 3);\nCREATE TABLE data.order_items_p4 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 8, REMAINDER 4);\nCREATE TABLE data.order_items_p5 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 8, REMAINDER 5);\nCREATE TABLE data.order_items_p6 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 8, REMAINDER 6);\nCREATE TABLE data.order_items_p7 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 8, REMAINDER 7);\n```\n\n### Generate Hash Partitions\n\n```sql\n-- Function to create N hash partitions\nCREATE FUNCTION private.create_hash_partitions(\n in_parent_table text,\n in_partition_count integer\n)\nRETURNS void\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_i integer;\n l_partition_name text;\nBEGIN\n FOR l_i IN 0..(in_partition_count - 1) LOOP\n l_partition_name := in_parent_table || '_p' || l_i;\n\n EXECUTE format(\n 'CREATE TABLE data.%I PARTITION OF data.%I FOR VALUES WITH (MODULUS %s, REMAINDER %s)',\n l_partition_name,\n in_parent_table,\n in_partition_count,\n l_i\n );\n END LOOP;\nEND;\n$;\n\n-- Usage\nSELECT private.create_hash_partitions('order_items', 16);\n```\n\n## Partition Management\n\n### Automatic Partition Creation (Time-Based)\n\n```sql\n-- Function to create next month's partition\nCREATE FUNCTION private.create_monthly_partition(\n in_table_name text,\n in_target_date date DEFAULT (date_trunc('month', now()) + interval '1 month')::date\n)\nRETURNS text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_partition_name text;\n l_start_date date;\n l_end_date date;\nBEGIN\n l_start_date := date_trunc('month', in_target_date);\n l_end_date := l_start_date + interval '1 month';\n l_partition_name := in_table_name || '_' || to_char(l_start_date, 'YYYY_MM');\n\n -- Check if partition already exists\n IF EXISTS (\n SELECT 1 FROM pg_tables\n WHERE schemaname = 'data' AND tablename = l_partition_name\n ) THEN\n RAISE NOTICE 'Partition % already exists', l_partition_name;\n RETURN l_partition_name;\n END IF;\n\n -- Create partition\n EXECUTE format(\n 'CREATE TABLE data.%I PARTITION OF data.%I FOR VALUES FROM (%L) TO (%L)',\n l_partition_name,\n in_table_name,\n l_start_date,\n l_end_date\n );\n\n RAISE NOTICE 'Created partition: data.%', l_partition_name;\n RETURN l_partition_name;\nEND;\n$;\n\n-- Schedule with pg_cron (run on 25th of each month)\nSELECT cron.schedule(\n 'create-events-partition',\n '0 0 25 * *',\n $SELECT private.create_monthly_partition('events')$\n);\n```\n\n### Partition Retention (Drop Old Partitions)\n\n```sql\n-- Function to drop partitions older than retention period\nCREATE FUNCTION private.drop_old_partitions(\n in_table_name text,\n in_retention_months integer\n)\nRETURNS integer\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_cutoff_date date;\n l_partition record;\n l_dropped_count integer := 0;\nBEGIN\n l_cutoff_date := date_trunc('month', now() - (in_retention_months || ' months')::interval);\n\n FOR l_partition IN\n SELECT\n c.relname AS partition_name,\n pg_get_expr(c.relpartbound, c.oid) AS bounds\n FROM pg_class p\n JOIN pg_inherits i ON p.oid = i.inhparent\n JOIN pg_class c ON i.inhrelid = c.oid\n WHERE p.relname = in_table_name\n AND p.relnamespace = 'data'::regnamespace\n AND c.relname != in_table_name || '_default'\n LOOP\n -- Extract start date from partition bounds\n -- Bounds format: FOR VALUES FROM ('2024-01-01') TO ('2024-02-01')\n IF l_partition.bounds ~ 'FROM \\(''(\\d{4}-\\d{2}-\\d{2})' THEN\n DECLARE\n l_partition_start date;\n BEGIN\n l_partition_start := (regexp_match(l_partition.bounds, 'FROM \\(''(\\d{4}-\\d{2}-\\d{2})'))[1]::date;\n\n IF l_partition_start \u003c l_cutoff_date THEN\n EXECUTE format('DROP TABLE data.%I', l_partition.partition_name);\n RAISE NOTICE 'Dropped partition: %', l_partition.partition_name;\n l_dropped_count := l_dropped_count + 1;\n END IF;\n END;\n END IF;\n END LOOP;\n\n RETURN l_dropped_count;\nEND;\n$;\n\n-- Keep 12 months of data\nSELECT private.drop_old_partitions('events', 12);\n\n-- Schedule monthly cleanup\nSELECT cron.schedule(\n 'cleanup-events-partitions',\n '0 2 1 * *', -- 2 AM on 1st of month\n $SELECT private.drop_old_partitions('events', 12)$\n);\n```\n\n### Detach Partition (Archive)\n\n```sql\n-- Detach partition for archiving (without dropping)\nALTER TABLE data.events DETACH PARTITION data.events_2023_01;\n\n-- Can now move to archive tablespace\nALTER TABLE data.events_2023_01 SET TABLESPACE archive_hdd;\n\n-- Or dump and remove\n-- pg_dump -t data.events_2023_01 > events_2023_01.sql\n-- DROP TABLE data.events_2023_01;\n\n-- Reattach if needed\nALTER TABLE data.events ATTACH PARTITION data.events_2023_01\n FOR VALUES FROM ('2023-01-01') TO ('2023-02-01');\n```\n\n### Detach Partition Concurrently (PostgreSQL 14+)\n\n```sql\n-- Non-blocking detach\nALTER TABLE data.events DETACH PARTITION data.events_2023_01 CONCURRENTLY;\n\n-- Note: Cannot be in a transaction block\n-- Note: Partition is still visible until detach completes\n```\n\n## Indexes on Partitioned Tables\n\n### Partition-Local Indexes\n\n```sql\n-- Index created on parent applies to all partitions\nCREATE INDEX events_event_type_idx ON data.events(event_type);\n\n-- Each partition gets its own copy of the index\n-- Verify:\nSELECT\n tablename,\n indexname\nFROM pg_indexes\nWHERE tablename LIKE 'events%'\nORDER BY tablename, indexname;\n```\n\n### Per-Partition Index Customization\n\n```sql\n-- Create index only on specific partition\nCREATE INDEX events_2024_01_payload_idx\n ON data.events_2024_01 USING gin(payload);\n\n-- Index on hot partition only\nCREATE INDEX events_active_customer_idx\n ON data.orders_active(customer_id);\n```\n\n### Unique Constraints with Partition Key\n\n```sql\n-- Unique constraint MUST include partition key\nCREATE TABLE data.users (\n id uuid NOT NULL DEFAULT uuidv7(),\n email text NOT NULL,\n region text NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now(),\n\n -- Works: includes partition key\n PRIMARY KEY (id, region),\n\n -- Works: includes partition key\n UNIQUE (email, region)\n) PARTITION BY LIST (region);\n\n-- Will NOT work:\n-- UNIQUE (email) -- Error: missing partition key\n```\n\n### Foreign Keys\n\n```sql\n-- Foreign key TO partitioned table (PostgreSQL 12+)\nCREATE TABLE data.order_items (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n order_id uuid NOT NULL,\n order_region text NOT NULL,\n product_id uuid NOT NULL,\n\n FOREIGN KEY (order_id, order_region)\n REFERENCES data.orders(id, region)\n ON DELETE CASCADE\n);\n\n-- Foreign key FROM partitioned table works normally\nCREATE TABLE data.events (\n id uuid NOT NULL DEFAULT uuidv7(),\n user_id uuid REFERENCES data.users_non_partitioned(id),\n created_at timestamptz NOT NULL,\n\n PRIMARY KEY (id, created_at)\n) PARTITION BY RANGE (created_at);\n```\n\n## Migrating to Partitioned Tables\n\n### Online Migration Strategy\n\n```sql\n-- Step 1: Create new partitioned table\nCREATE TABLE data.events_new (\n id uuid NOT NULL DEFAULT uuidv7(),\n event_type text NOT NULL,\n payload jsonb NOT NULL DEFAULT '{}',\n created_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (id, created_at)\n) PARTITION BY RANGE (created_at);\n\n-- Create partitions covering existing data\nCREATE TABLE data.events_new_2023_01 PARTITION OF data.events_new\n FOR VALUES FROM ('2023-01-01') TO ('2023-02-01');\n-- ... create all needed partitions\n\nCREATE TABLE data.events_new_default PARTITION OF data.events_new DEFAULT;\n\n-- Step 2: Create trigger to dual-write\nCREATE FUNCTION private.events_dual_write_trigger()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nBEGIN\n IF TG_OP = 'INSERT' THEN\n INSERT INTO data.events_new VALUES (NEW.*);\n ELSIF TG_OP = 'UPDATE' THEN\n UPDATE data.events_new SET\n event_type = NEW.event_type,\n payload = NEW.payload,\n created_at = NEW.created_at\n WHERE id = NEW.id AND created_at = NEW.created_at;\n ELSIF TG_OP = 'DELETE' THEN\n DELETE FROM data.events_new\n WHERE id = OLD.id AND created_at = OLD.created_at;\n END IF;\n RETURN NULL;\nEND;\n$;\n\nCREATE TRIGGER events_dual_write_trg\n AFTER INSERT OR UPDATE OR DELETE ON data.events\n FOR EACH ROW EXECUTE FUNCTION private.events_dual_write_trigger();\n\n-- Step 3: Backfill historical data (in batches)\nINSERT INTO data.events_new\nSELECT * FROM data.events\nWHERE created_at \u003c '2024-01-01' -- Before trigger was enabled\nON CONFLICT DO NOTHING;\n\n-- Step 4: Verify row counts match\nSELECT\n (SELECT count(*) FROM data.events) AS old_count,\n (SELECT count(*) FROM data.events_new) AS new_count;\n\n-- Step 5: Swap tables (brief lock)\nBEGIN;\nDROP TRIGGER events_dual_write_trg ON data.events;\nALTER TABLE data.events RENAME TO events_old;\nALTER TABLE data.events_new RENAME TO events;\nCOMMIT;\n\n-- Step 6: Update dependent objects (views, functions)\n-- Then drop old table when confident\n-- DROP TABLE data.events_old;\n```\n\n### Convert Existing Table to Partition\n\n```sql\n-- Convert existing table to become a partition of new parent\n-- (When existing table matches desired partition range)\n\n-- Step 1: Create partitioned parent with same structure\nCREATE TABLE data.events_partitioned (\n LIKE data.events INCLUDING ALL\n) PARTITION BY RANGE (created_at);\n\n-- Step 2: Attach existing table as partition\nALTER TABLE data.events_partitioned\n ATTACH PARTITION data.events\n FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');\n\n-- Note: This validates all rows match the partition constraint\n-- For large tables, use:\nALTER TABLE data.events ADD CONSTRAINT events_partition_check\n CHECK (created_at >= '2024-01-01' AND created_at \u003c '2024-02-01')\n NOT VALID;\nALTER TABLE data.events VALIDATE CONSTRAINT events_partition_check;\n\n-- Then attach (validation is skipped because constraint exists)\nALTER TABLE data.events_partitioned\n ATTACH PARTITION data.events\n FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');\n```\n\n## Performance Optimization\n\n### Partition Pruning\n\n```sql\n-- Enable partition pruning (on by default)\nSET enable_partition_pruning = on;\n\n-- Query that benefits from pruning\nEXPLAIN (ANALYZE)\nSELECT * FROM data.events\nWHERE created_at >= '2024-03-01' AND created_at \u003c '2024-03-15';\n-- Shows: Scans only events_2024_03 partition\n\n-- Pruning works with prepared statements (PostgreSQL 11+)\nPREPARE events_query(timestamptz, timestamptz) AS\n SELECT * FROM data.events WHERE created_at >= $1 AND created_at \u003c $2;\n\nEXECUTE events_query('2024-03-01', '2024-03-15');\n```\n\n### Partition-Wise Aggregation\n\n```sql\n-- Enable partition-wise aggregation\nSET enable_partitionwise_aggregate = on;\n\n-- Aggregates run in parallel per partition\nEXPLAIN (ANALYZE)\nSELECT date_trunc('day', created_at), count(*)\nFROM data.events\nWHERE created_at >= '2024-01-01'\nGROUP BY 1;\n-- Shows: Partial aggregates per partition, then combine\n```\n\n### Partition-Wise Joins\n\n```sql\n-- Enable partition-wise joins\nSET enable_partitionwise_join = on;\n\n-- Both tables partitioned same way\nCREATE TABLE data.event_details (\n event_id uuid NOT NULL,\n created_at timestamptz NOT NULL,\n details jsonb,\n\n PRIMARY KEY (event_id, created_at)\n) PARTITION BY RANGE (created_at);\n\n-- Join can be done per-partition\nEXPLAIN (ANALYZE)\nSELECT e.*, d.details\nFROM data.events e\nJOIN data.event_details d ON e.id = d.event_id AND e.created_at = d.created_at\nWHERE e.created_at >= '2024-03-01' AND e.created_at \u003c '2024-04-01';\n```\n\n### Monitor Partition Usage\n\n```sql\n-- Check partition sizes\nSELECT\n c.relname AS partition_name,\n pg_size_pretty(pg_relation_size(c.oid)) AS size,\n pg_size_pretty(pg_total_relation_size(c.oid)) AS total_size\nFROM pg_class p\nJOIN pg_inherits i ON p.oid = i.inhparent\nJOIN pg_class c ON i.inhrelid = c.oid\nWHERE p.relname = 'events'\n AND p.relnamespace = 'data'::regnamespace\nORDER BY pg_relation_size(c.oid) DESC;\n\n-- Check partition row counts\nSELECT\n c.relname AS partition_name,\n c.reltuples::bigint AS estimated_rows\nFROM pg_class p\nJOIN pg_inherits i ON p.oid = i.inhparent\nJOIN pg_class c ON i.inhrelid = c.oid\nWHERE p.relname = 'events'\n AND p.relnamespace = 'data'::regnamespace\nORDER BY c.reltuples DESC;\n```\n\n## Common Patterns\n\n### Sub-Partitioning\n\n```sql\n-- Two-level partitioning: first by region, then by date\nCREATE TABLE data.sales (\n id uuid NOT NULL DEFAULT uuidv7(),\n region text NOT NULL,\n amount numeric(15,2) NOT NULL,\n sale_date date NOT NULL,\n\n PRIMARY KEY (id, region, sale_date)\n) PARTITION BY LIST (region);\n\n-- Create region partitions\nCREATE TABLE data.sales_us PARTITION OF data.sales\n FOR VALUES IN ('US')\n PARTITION BY RANGE (sale_date);\n\nCREATE TABLE data.sales_eu PARTITION OF data.sales\n FOR VALUES IN ('EU')\n PARTITION BY RANGE (sale_date);\n\n-- Create date sub-partitions\nCREATE TABLE data.sales_us_2024_q1 PARTITION OF data.sales_us\n FOR VALUES FROM ('2024-01-01') TO ('2024-04-01');\nCREATE TABLE data.sales_us_2024_q2 PARTITION OF data.sales_us\n FOR VALUES FROM ('2024-04-01') TO ('2024-07-01');\n\nCREATE TABLE data.sales_eu_2024_q1 PARTITION OF data.sales_eu\n FOR VALUES FROM ('2024-01-01') TO ('2024-04-01');\nCREATE TABLE data.sales_eu_2024_q2 PARTITION OF data.sales_eu\n FOR VALUES FROM ('2024-04-01') TO ('2024-07-01');\n```\n\n### Default Partition Handling\n\n```sql\n-- Always create default partition for unexpected values\nCREATE TABLE data.events_default PARTITION OF data.events DEFAULT;\n\n-- Monitor default partition (should stay empty/small)\nSELECT count(*) FROM data.events_default;\n\n-- Split out rows from default when creating new partition\n-- Step 1: Create new partition\nCREATE TABLE data.events_2024_04 PARTITION OF data.events\n FOR VALUES FROM ('2024-04-01') TO ('2024-05-01');\n-- If default has matching rows, they're automatically moved\n\n-- Or explicitly handle before creating:\nBEGIN;\n-- Detach default\nALTER TABLE data.events DETACH PARTITION data.events_default;\n\n-- Create new partition\nCREATE TABLE data.events_2024_04 PARTITION OF data.events\n FOR VALUES FROM ('2024-04-01') TO ('2024-05-01');\n\n-- Move matching rows\nINSERT INTO data.events_2024_04\nSELECT * FROM data.events_default\nWHERE created_at >= '2024-04-01' AND created_at \u003c '2024-05-01';\n\nDELETE FROM data.events_default\nWHERE created_at >= '2024-04-01' AND created_at \u003c '2024-05-01';\n\n-- Reattach default\nALTER TABLE data.events ATTACH PARTITION data.events_default DEFAULT;\nCOMMIT;\n```\n\n### API Functions for Partitioned Tables\n\n```sql\n-- Select function with partition pruning hint\nCREATE FUNCTION api.select_events(\n in_start_date timestamptz,\n in_end_date timestamptz,\n in_event_type text DEFAULT NULL,\n in_limit integer DEFAULT 100\n)\nRETURNS TABLE (\n id uuid,\n event_type text,\n payload jsonb,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, event_type, payload, created_at\n FROM data.events\n WHERE created_at >= in_start_date\n AND created_at \u003c in_end_date\n AND (in_event_type IS NULL OR event_type = in_event_type)\n ORDER BY created_at DESC\n LIMIT in_limit;\n$;\n\n-- Insert procedure (partition selected automatically)\nCREATE PROCEDURE api.insert_event(\n in_event_type text,\n in_payload jsonb,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.events (event_type, payload)\n VALUES (in_event_type, in_payload)\n RETURNING id INTO io_id;\nEND;\n$;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":23821,"content_sha256":"15ddb68f51e834cbe1e26d61cddd1f0f97e0d281371203214da8d92a007fc7fa"},{"filename":"references/performance-tuning.md","content":"# Performance Tuning Patterns\n\nThis document covers query optimization, EXPLAIN analysis, connection pooling, and partitioning strategies for PostgreSQL.\n\n## Table of Contents\n\n1. [EXPLAIN ANALYZE Guide](#explain-analyze-guide)\n2. [Common Query Optimizations](#common-query-optimizations)\n3. [Index Optimization](#index-optimization)\n4. [Connection Pooling](#connection-pooling)\n5. [Prepared Statements](#prepared-statements)\n6. [Partitioning Strategies](#partitioning-strategies)\n7. [Configuration Tuning](#configuration-tuning)\n8. [Monitoring Queries](#monitoring-queries)\n\n## EXPLAIN ANALYZE Guide\n\n### Basic Usage\n\n```sql\n-- Always use ANALYZE for actual execution times\nEXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)\nSELECT * FROM data.orders WHERE customer_id = 'uuid-here';\n\n-- For production-safe analysis (no actual execution)\nEXPLAIN (COSTS, FORMAT TEXT)\nSELECT * FROM data.orders WHERE customer_id = 'uuid-here';\n```\n\n### Reading EXPLAIN Output\n\n```\n QUERY PLAN\n------------------------------------------------------------------------------------------------------------------------------\n Index Scan using orders_customer_id_idx on orders (cost=0.43..8.45 rows=1 width=100) (actual time=0.025..0.027 rows=1 loops=1)\n Index Cond: (customer_id = 'uuid-here'::uuid)\n Buffers: shared hit=3\n Planning Time: 0.085 ms\n Execution Time: 0.045 ms\n```\n\n### Key Metrics to Watch\n\n| Metric | Meaning | Concern Level |\n|--------|---------|---------------|\n| `Seq Scan` | Full table scan | ⚠️ Bad for large tables |\n| `Index Scan` | Using B-tree index | ✅ Good |\n| `Index Only Scan` | Data from index alone | ✅ Excellent |\n| `Bitmap Heap Scan` | Multiple index conditions | ✅ Good for OR conditions |\n| `Nested Loop` | Row-by-row join | ⚠️ Watch row counts |\n| `Hash Join` | Hash table for join | ✅ Good for larger datasets |\n| `Merge Join` | Sorted merge | ✅ Good for sorted data |\n| `rows=X` vs `actual rows=Y` | Estimate accuracy | ⚠️ If very different, run ANALYZE |\n\n### Common Problems and Solutions\n\n#### Problem: Sequential Scan on Large Table\n\n```sql\n-- Bad: Full table scan\nEXPLAIN ANALYZE\nSELECT * FROM data.orders WHERE status = 'pending';\n\n-- Shows: Seq Scan on orders (cost=0.00..1234.00 rows=50000 ...)\n\n-- Solution: Add index\nCREATE INDEX orders_status_idx ON data.orders(status);\n\n-- Or partial index if status has few values\nCREATE INDEX orders_pending_idx ON data.orders(created_at)\n WHERE status = 'pending';\n```\n\n#### Problem: Wrong Index Being Used\n\n```sql\n-- Check which indexes exist\nSELECT indexname, indexdef \nFROM pg_indexes \nWHERE tablename = 'orders';\n\n-- Force index usage for testing (not for production)\nSET enable_seqscan = off;\nEXPLAIN ANALYZE SELECT ...;\nSET enable_seqscan = on;\n\n-- Update statistics if estimates are wrong\nANALYZE data.orders;\n```\n\n#### Problem: Slow Join\n\n```sql\n-- Bad: Nested loop on large tables\nEXPLAIN ANALYZE\nSELECT o.*, c.name\nFROM data.orders o\nJOIN data.customers c ON c.id = o.customer_id\nWHERE o.created_at > '2024-01-01';\n\n-- Solution 1: Ensure FK is indexed\nCREATE INDEX orders_customer_id_idx ON data.orders(customer_id);\n\n-- Solution 2: Use covering index\nCREATE INDEX orders_created_customer_idx\n ON data.orders(created_at, customer_id);\n```\n\n### EXPLAIN Format Options\n\n```sql\n-- Text (default, human readable)\nEXPLAIN (FORMAT TEXT) SELECT ...;\n\n-- JSON (for programmatic analysis)\nEXPLAIN (FORMAT JSON) SELECT ...;\n\n-- YAML\nEXPLAIN (FORMAT YAML) SELECT ...;\n\n-- Full analysis with all options\nEXPLAIN (\n ANALYZE, -- Actually execute\n BUFFERS, -- Show buffer usage\n COSTS, -- Show cost estimates\n TIMING, -- Show actual timing\n VERBOSE, -- Show extra info\n FORMAT TEXT\n) SELECT ...;\n```\n\n## Common Query Optimizations\n\n### Pagination Optimization\n\n```sql\n-- Bad: OFFSET for deep pagination\nSELECT * FROM data.orders \nORDER BY created_at DESC \nLIMIT 20 OFFSET 10000; -- Scans 10020 rows!\n\n-- Good: Keyset pagination (cursor-based)\nSELECT * FROM data.orders \nWHERE created_at \u003c $last_seen_created_at\nORDER BY created_at DESC \nLIMIT 20;\n\n-- API function with keyset pagination\nCREATE FUNCTION api.select_orders_paginated(\n in_cursor timestamptz DEFAULT NULL,\n in_limit integer DEFAULT 20\n)\nRETURNS TABLE (\n id uuid,\n created_at timestamptz,\n total numeric,\n next_cursor timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n WITH page AS (\n SELECT id, created_at, total\n FROM data.orders\n WHERE (in_cursor IS NULL OR created_at \u003c in_cursor)\n ORDER BY created_at DESC\n LIMIT in_limit + 1 -- Fetch one extra to detect more pages\n )\n SELECT \n id,\n created_at,\n total,\n CASE \n WHEN ROW_NUMBER() OVER () > in_limit THEN created_at\n ELSE NULL\n END AS next_cursor\n FROM page\n LIMIT in_limit;\n$;\n```\n\n### Avoiding N+1 Queries\n\n```sql\n-- Bad: Called in a loop from application\nSELECT * FROM data.orders WHERE customer_id = $1;\n\n-- Good: Batch fetch\nCREATE FUNCTION api.select_orders_by_customers(in_customer_ids uuid[])\nRETURNS TABLE (\n customer_id uuid,\n order_id uuid,\n total numeric\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT customer_id, id, total\n FROM data.orders\n WHERE customer_id = ANY(in_customer_ids)\n ORDER BY customer_id, created_at DESC;\n$;\n```\n\n### Optimizing COUNT Queries\n\n```sql\n-- Bad: Exact count on large table\nSELECT COUNT(*) FROM data.orders WHERE status = 'pending';\n\n-- Good: Approximate count (very fast)\nSELECT reltuples::bigint AS estimate\nFROM pg_class\nWHERE relname = 'orders';\n\n-- Good: Exact count with limit check\nCREATE FUNCTION api.count_orders_limited(\n in_status text,\n in_max integer DEFAULT 1000\n)\nRETURNS integer\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT COUNT(*)::integer\n FROM (\n SELECT 1 FROM data.orders \n WHERE status = in_status \n LIMIT in_max\n ) sub;\n$;\n```\n\n### Conditional Aggregation\n\n```sql\n-- Bad: Multiple queries\nSELECT COUNT(*) FROM data.orders WHERE status = 'pending';\nSELECT COUNT(*) FROM data.orders WHERE status = 'shipped';\nSELECT COUNT(*) FROM data.orders WHERE status = 'delivered';\n\n-- Good: Single query with conditional aggregation\nSELECT \n COUNT(*) FILTER (WHERE status = 'pending') AS pending_count,\n COUNT(*) FILTER (WHERE status = 'shipped') AS shipped_count,\n COUNT(*) FILTER (WHERE status = 'delivered') AS delivered_count,\n SUM(total) FILTER (WHERE status = 'delivered') AS delivered_total\nFROM data.orders\nWHERE created_at > now() - interval '30 days';\n```\n\n### EXISTS vs IN vs JOIN\n\n```sql\n-- Use EXISTS for existence checks (usually fastest)\nSELECT * FROM data.customers c\nWHERE EXISTS (\n SELECT 1 FROM data.orders o \n WHERE o.customer_id = c.id \n AND o.status = 'pending'\n);\n\n-- Use IN for small, known lists\nSELECT * FROM data.orders\nWHERE status IN ('pending', 'processing', 'shipped');\n\n-- Avoid NOT IN with NULLs (use NOT EXISTS)\n-- Bad: Returns no rows if subquery has NULLs\nSELECT * FROM data.customers\nWHERE id NOT IN (SELECT customer_id FROM data.orders);\n\n-- Good: Handles NULLs correctly\nSELECT * FROM data.customers c\nWHERE NOT EXISTS (\n SELECT 1 FROM data.orders o WHERE o.customer_id = c.id\n);\n```\n\n## Index Optimization\n\n### Composite Index Column Order\n\n```sql\n-- Rule: Most selective column first, range/sort column last\n\n-- Query: WHERE status = 'pending' AND created_at > '2024-01-01'\n-- Good: equality column first\nCREATE INDEX orders_status_created_idx\n ON data.orders(status, created_at);\n\n-- Query: WHERE customer_id = $1 ORDER BY created_at DESC\n-- Good: equality first, sort last\nCREATE INDEX orders_customer_created_idx\n ON data.orders(customer_id, created_at DESC);\n```\n\n### Covering Indexes\n\n```sql\n-- Query frequently needs id, total, status\n-- Include extra columns to avoid table lookup\nCREATE INDEX orders_customer_covering_idx\n ON data.orders(customer_id)\n INCLUDE (total, status, created_at);\n\n-- Results in \"Index Only Scan\" - much faster\n```\n\n### Partial Indexes\n\n```sql\n-- Index only active customers (90% of queries)\nCREATE INDEX customers_email_active_idx\n ON data.customers(email)\n WHERE is_active = true;\n\n-- Index only recent orders\nCREATE INDEX orders_pending_recent_idx\n ON data.orders(created_at)\n WHERE status = 'pending'\n AND created_at > '2024-01-01';\n\n-- Much smaller index, faster updates\n```\n\n### Expression Indexes\n\n```sql\n-- Query: WHERE lower(email) = lower($1)\nCREATE INDEX customers_email_lower_idx\n ON data.customers(lower(email));\n\n-- Query: WHERE date_trunc('day', created_at) = $1\nCREATE INDEX orders_created_day_idx\n ON data.orders(date_trunc('day', created_at));\n\n-- Query: WHERE (data->>'category') = $1\nCREATE INDEX products_category_idx\n ON data.products((data->>'category'));\n```\n\n### Index Maintenance\n\n```sql\n-- Check index usage\nSELECT \n schemaname,\n tablename,\n indexname,\n idx_scan,\n idx_tup_read,\n idx_tup_fetch\nFROM pg_stat_user_indexes\nWHERE schemaname = 'data'\nORDER BY idx_scan;\n\n-- Find unused indexes (candidates for removal)\nSELECT \n schemaname || '.' || tablename AS table,\n indexname,\n pg_size_pretty(pg_relation_size(indexrelid)) AS size\nFROM pg_stat_user_indexes\nWHERE idx_scan = 0\n AND schemaname = 'data'\nORDER BY pg_relation_size(indexrelid) DESC;\n\n-- Rebuild bloated indexes\nREINDEX INDEX CONCURRENTLY orders_customer_id_idx;\n\n-- Or rebuild all indexes on a table\nREINDEX TABLE CONCURRENTLY data.orders;\n```\n\n## Connection Pooling\n\n### PgBouncer with SECURITY DEFINER\n\nWhen using connection pooling with `SECURITY DEFINER` functions, the functions execute as the function owner, not the pooled connection user. This is actually ideal for the Table API pattern.\n\n```ini\n# pgbouncer.ini\n[databases]\nmyapp = host=localhost dbname=myapp\n\n[pgbouncer]\nlisten_addr = 127.0.0.1\nlisten_port = 6432\nauth_type = md5\nauth_file = /etc/pgbouncer/userlist.txt\npool_mode = transaction # Best for SECURITY DEFINER\nmax_client_conn = 1000\ndefault_pool_size = 20\n```\n\n### Session Variables with Pooling\n\n```sql\n-- Problem: SET variables don't persist across pooled connections\n\n-- Solution: Pass context as parameters\nCREATE FUNCTION api.get_my_orders(in_user_id uuid)\nRETURNS TABLE (...)\nAS $ ... $;\n\n-- Or use transaction-local settings\nCREATE FUNCTION api.set_context(in_user_id uuid)\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n PERFORM set_config('myapp.user_id', in_user_id::text, true); -- true = local to transaction\nEND;\n$;\n\n-- Application calls at start of each transaction:\n-- BEGIN;\n-- SELECT api.set_context('user-uuid');\n-- SELECT * FROM api.get_my_orders();\n-- COMMIT;\n```\n\n### Pool Mode Recommendations\n\n| Pool Mode | Use Case | SECURITY DEFINER Compatible |\n|-----------|----------|----------------------------|\n| `session` | Long-lived connections, session variables | ✅ Yes |\n| `transaction` | Short queries, web apps | ✅ Yes (recommended) |\n| `statement` | Simple queries only | ⚠️ Limited |\n\n## Prepared Statements\n\nPrepared statements separate query parsing/planning from execution. They improve performance for frequently executed queries by reusing the query plan.\n\n### Basic PREPARE / EXECUTE\n\n```sql\n-- Prepare a named statement\nPREPARE get_customer_orders (uuid) AS\n SELECT id, total, created_at\n FROM data.orders\n WHERE customer_id = $1\n ORDER BY created_at DESC;\n\n-- Execute with parameters (plan is reused)\nEXECUTE get_customer_orders('550e8400-e29b-41d4-a716-446655440000');\n\n-- Deallocate when done\nDEALLOCATE get_customer_orders;\n\n-- Deallocate all prepared statements\nDEALLOCATE ALL;\n```\n\n### When to Use\n\nPrepared statements help when:\n- The same query structure is executed hundreds or thousands of times per session\n- The query has a stable plan regardless of parameter values\n- You are using session-mode pooling or persistent connections\n\nThey add overhead when:\n- A query is executed only once (parsing + planning cost is paid regardless, plus the extra round trip)\n- Parameter values cause dramatically different optimal plans (e.g., highly skewed distributions)\n\n### Plan Caching: Generic vs Custom Plans\n\nPostgreSQL creates **custom plans** (parameter-specific) for the first 5 executions, then switches to a **generic plan** if it performs comparably. You can control this behavior.\n\n```sql\n-- Force generic plans (skip the 5 custom-plan warm-up)\nSET plan_cache_mode = 'force_generic_plan';\n\n-- Force custom plans (always re-plan with actual parameter values)\nSET plan_cache_mode = 'force_custom_plan';\n\n-- Default: auto-select (recommended for most workloads)\nSET plan_cache_mode = 'auto';\n```\n\nUse `force_custom_plan` when parameter values produce very different result set sizes (e.g., `status = 'active'` returns 95% of rows vs `status = 'deleted'` returns 0.1%).\n\n### Connection Pooling Interaction\n\nPrepared statements are **per-connection state**. In transaction-mode pooling (the recommended mode for the Table API pattern), the server connection changes between transactions, so prepared statements are lost.\n\n```ini\n# pgbouncer.ini — clean up prepared statements when connection returns to pool\nserver_reset_query = DEALLOCATE ALL; DISCARD ALL\n```\n\n**Best approach**: Prefer server-side functions (Table API) over client-side prepared statements. When your application calls `SELECT * FROM api.get_customer(in_id := $1)`, PostgreSQL caches the plan for the function body automatically — no client-side `PREPARE` needed, and it works with any pool mode.\n\n```sql\n-- Table API function — plan is cached server-side, pool-mode safe\nCREATE FUNCTION api.get_customer(in_id uuid)\nRETURNS TABLE (id uuid, email text, name text)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name FROM data.customers WHERE id = in_id;\n$;\n```\n\nIf you must use client-side prepared statements with PgBouncer 1.21+, enable `protocol_query` mode which proxies the PostgreSQL extended query protocol (Parse/Bind/Execute) and handles prepared statement forwarding transparently.\n\n### Monitoring Prepared Statements\n\n```sql\n-- View all prepared statements in the current session\nSELECT name, statement, prepare_time, parameter_types, result_types\nFROM pg_prepared_statements;\n\n-- Check if generic or custom plan is in use\n-- (generic_plans > 0 indicates the planner switched to generic)\nSELECT name, generic_plans, custom_plans\nFROM pg_prepared_statements;\n```\n\n## Partitioning Strategies\n\n### Range Partitioning (Time-Based)\n\n```sql\n-- Create partitioned table\nCREATE TABLE data.events (\n id uuid NOT NULL DEFAULT uuidv7(),\n event_type text NOT NULL,\n payload jsonb,\n created_at timestamptz NOT NULL DEFAULT now()\n) PARTITION BY RANGE (created_at);\n\n-- Create partitions\nCREATE TABLE data.events_2024_01 PARTITION OF data.events\n FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');\nCREATE TABLE data.events_2024_02 PARTITION OF data.events\n FOR VALUES FROM ('2024-02-01') TO ('2024-03-01');\n-- ... more partitions\n\n-- Create default partition for unexpected dates\nCREATE TABLE data.events_default PARTITION OF data.events DEFAULT;\n\n-- Indexes are created per-partition\nCREATE INDEX events_2024_01_type_idx ON data.events_2024_01(event_type);\nCREATE INDEX events_2024_02_type_idx ON data.events_2024_02(event_type);\n```\n\n### Automatic Partition Management\n\n```sql\n-- Function to create next month's partition\nCREATE OR REPLACE FUNCTION private.create_next_event_partition()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_start_date date;\n l_end_date date;\n l_partition_name text;\nBEGIN\n l_start_date := date_trunc('month', now() + interval '1 month');\n l_end_date := l_start_date + interval '1 month';\n l_partition_name := 'events_' || to_char(l_start_date, 'YYYY_MM');\n \n -- Check if partition exists\n IF NOT EXISTS (\n SELECT 1 FROM pg_tables \n WHERE schemaname = 'data' AND tablename = l_partition_name\n ) THEN\n EXECUTE format(\n 'CREATE TABLE data.%I PARTITION OF data.events \n FOR VALUES FROM (%L) TO (%L)',\n l_partition_name, l_start_date, l_end_date\n );\n \n EXECUTE format(\n 'CREATE INDEX %s_type_idx ON data.%I(event_type)',\n l_partition_name, l_partition_name\n );\n \n RAISE NOTICE 'Created partition: %', l_partition_name;\n END IF;\nEND;\n$;\n\n-- Schedule with pg_cron\nSELECT cron.schedule('create-event-partitions', '0 0 25 * *', \n 'SELECT private.create_next_event_partition()');\n```\n\n### List Partitioning (By Category)\n\n```sql\n-- Partition by region\nCREATE TABLE data.customers (\n id uuid NOT NULL DEFAULT uuidv7(),\n email text NOT NULL,\n region text NOT NULL,\n name text\n) PARTITION BY LIST (region);\n\nCREATE TABLE data.customers_us PARTITION OF data.customers\n FOR VALUES IN ('us-east', 'us-west', 'us-central');\nCREATE TABLE data.customers_eu PARTITION OF data.customers\n FOR VALUES IN ('eu-west', 'eu-central', 'eu-north');\nCREATE TABLE data.customers_apac PARTITION OF data.customers\n FOR VALUES IN ('apac-east', 'apac-south');\n```\n\n### Hash Partitioning (Even Distribution)\n\n```sql\n-- Distribute by customer_id hash\nCREATE TABLE data.order_items (\n id uuid NOT NULL DEFAULT uuidv7(),\n order_id uuid NOT NULL,\n customer_id uuid NOT NULL,\n product_id uuid NOT NULL,\n quantity integer NOT NULL\n) PARTITION BY HASH (customer_id);\n\n-- Create 4 partitions\nCREATE TABLE data.order_items_p0 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 4, REMAINDER 0);\nCREATE TABLE data.order_items_p1 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 4, REMAINDER 1);\nCREATE TABLE data.order_items_p2 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 4, REMAINDER 2);\nCREATE TABLE data.order_items_p3 PARTITION OF data.order_items\n FOR VALUES WITH (MODULUS 4, REMAINDER 3);\n```\n\n## Configuration Tuning\n\n### Memory Settings\n\n```sql\n-- Check current settings\nSHOW shared_buffers; -- RAM for caching (25% of RAM typical)\nSHOW effective_cache_size; -- Estimate of OS cache (50-75% of RAM)\nSHOW work_mem; -- Per-operation sort memory (4-64MB typical)\nSHOW maintenance_work_mem; -- For VACUUM, CREATE INDEX (256MB-1GB)\n\n-- Recommended starting points (adjust for your workload)\n-- In postgresql.conf:\n-- shared_buffers = 4GB # 25% of 16GB RAM\n-- effective_cache_size = 12GB # 75% of 16GB RAM\n-- work_mem = 64MB # For complex queries\n-- maintenance_work_mem = 512MB # For VACUUM/REINDEX\n```\n\n### Write Performance\n\n```sql\n-- For write-heavy workloads\n-- wal_buffers = 64MB\n-- checkpoint_completion_target = 0.9\n-- max_wal_size = 4GB\n\n-- For batch inserts (temporarily)\nSET synchronous_commit = off; -- Caution: risk of data loss\nSET wal_level = minimal; -- Requires restart, reduces WAL\n```\n\n### Query Planner\n\n```sql\n-- Check planner settings\nSHOW random_page_cost; -- SSD: 1.1, HDD: 4.0\nSHOW effective_io_concurrency; -- SSD: 200, HDD: 2\n\n-- Update for SSD storage\nALTER SYSTEM SET random_page_cost = 1.1;\nALTER SYSTEM SET effective_io_concurrency = 200;\nSELECT pg_reload_conf();\n```\n\n## JIT Compilation\n\n### Overview\n\nJIT (Just-In-Time) compilation can speed up CPU-intensive queries by compiling expressions and tuple deforming into native code. Available since PostgreSQL 11.\n\n### When JIT Helps\n\n```sql\n-- JIT beneficial for:\n-- - Complex expressions in WHERE, SELECT\n-- - Large table scans with many columns\n-- - Aggregations over many rows\n-- - Queries spending significant time in expression evaluation\n\n-- Check if JIT is available\nSELECT name, setting FROM pg_settings WHERE name LIKE 'jit%';\n```\n\n### JIT Settings\n\n```sql\n-- Enable/disable JIT (default: on in PG12+)\nSET jit = on;\n\n-- Cost thresholds (query must exceed these costs)\nSET jit_above_cost = 100000; -- Enable JIT (default: 100000)\nSET jit_inline_above_cost = 500000; -- Inline functions (default: 500000)\nSET jit_optimize_above_cost = 500000; -- Full optimization (default: 500000)\n\n-- For OLAP/analytics workloads, lower thresholds\nALTER SYSTEM SET jit_above_cost = 10000;\nALTER SYSTEM SET jit_inline_above_cost = 50000;\nALTER SYSTEM SET jit_optimize_above_cost = 50000;\nSELECT pg_reload_conf();\n```\n\n### Monitoring JIT Usage\n\n```sql\n-- Check JIT in EXPLAIN\nEXPLAIN (ANALYZE, BUFFERS)\nSELECT sum(total), avg(total), count(*)\nFROM data.large_orders\nWHERE status = 'completed';\n\n-- Look for:\n-- JIT:\n-- Functions: 5\n-- Options: Inlining true, Optimization true, Expressions true, Deforming true\n-- Timing: Generation 1.234 ms, Inlining 5.678 ms, Optimization 12.345 ms, Emission 23.456 ms, Total 42.713 ms\n```\n\n### When to Disable JIT\n\n```sql\n-- JIT adds overhead for compilation\n-- Disable for short OLTP queries\n\n-- Session level\nSET jit = off;\n\n-- Or raise thresholds for mixed workloads\nSET jit_above_cost = 500000;\n\n-- In application connection\n-- postgresql://user:pass@host/db?options=-c%20jit=off\n```\n\n### JIT Troubleshooting\n\n```sql\n-- JIT not being used when expected?\n-- 1. Check if enabled\nSHOW jit;\n\n-- 2. Check if LLVM is installed\nSELECT pg_jit_available();\n\n-- 3. Check query cost exceeds threshold\nEXPLAIN (COSTS) SELECT ...;\n-- Total cost must exceed jit_above_cost\n\n-- JIT slowing down queries?\n-- Compilation time can exceed execution savings for small result sets\n-- Solution: Raise thresholds or disable for that query\nSET LOCAL jit = off;\n```\n\n## Monitoring Queries\n\n### Slow Query Detection\n\n```sql\n-- Enable slow query logging in postgresql.conf\n-- log_min_duration_statement = 1000 # Log queries > 1 second\n\n-- Find slow queries with pg_stat_statements\nCREATE EXTENSION IF NOT EXISTS pg_stat_statements;\n\nSELECT \n round(total_exec_time::numeric, 2) AS total_time_ms,\n calls,\n round(mean_exec_time::numeric, 2) AS avg_time_ms,\n round((100 * total_exec_time / sum(total_exec_time) OVER ())::numeric, 2) AS percent_total,\n query\nFROM pg_stat_statements\nORDER BY total_exec_time DESC\nLIMIT 20;\n```\n\n### Table Statistics\n\n```sql\n-- Table sizes\nSELECT \n schemaname,\n tablename,\n pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) AS total_size,\n pg_size_pretty(pg_relation_size(schemaname || '.' || tablename)) AS table_size,\n pg_size_pretty(pg_indexes_size(schemaname || '.' || tablename)) AS index_size\nFROM pg_tables\nWHERE schemaname = 'data'\nORDER BY pg_total_relation_size(schemaname || '.' || tablename) DESC;\n\n-- Table activity\nSELECT \n schemaname,\n relname,\n seq_scan,\n seq_tup_read,\n idx_scan,\n idx_tup_fetch,\n n_tup_ins,\n n_tup_upd,\n n_tup_del,\n n_live_tup,\n n_dead_tup,\n last_vacuum,\n last_autovacuum,\n last_analyze\nFROM pg_stat_user_tables\nWHERE schemaname = 'data'\nORDER BY seq_scan DESC;\n```\n\n### Lock Monitoring\n\n```sql\n-- Current locks\nSELECT \n l.pid,\n l.mode,\n l.granted,\n a.usename,\n a.query,\n a.state,\n a.wait_event_type,\n a.wait_event\nFROM pg_locks l\nJOIN pg_stat_activity a ON l.pid = a.pid\nWHERE l.relation IS NOT NULL\nORDER BY l.granted, l.pid;\n\n-- Blocking queries\nSELECT \n blocked_locks.pid AS blocked_pid,\n blocked_activity.usename AS blocked_user,\n blocking_locks.pid AS blocking_pid,\n blocking_activity.usename AS blocking_user,\n blocked_activity.query AS blocked_statement,\n blocking_activity.query AS blocking_statement\nFROM pg_catalog.pg_locks blocked_locks\nJOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid\nJOIN pg_catalog.pg_locks blocking_locks \n ON blocking_locks.locktype = blocked_locks.locktype\n AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database\n AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation\n AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page\n AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple\n AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid\n AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid\n AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid\n AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid\n AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid\n AND blocking_locks.pid != blocked_locks.pid\nJOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid\nWHERE NOT blocked_locks.granted;\n```\n\n### Custom Performance Views\n\n```sql\n-- Create view for easy monitoring\nCREATE OR REPLACE VIEW api.v_performance_stats AS\nSELECT \n 'table_stats' AS category,\n jsonb_build_object(\n 'total_tables', (SELECT COUNT(*) FROM pg_tables WHERE schemaname = 'data'),\n 'largest_table', (\n SELECT tablename FROM pg_tables \n WHERE schemaname = 'data'\n ORDER BY pg_total_relation_size(schemaname || '.' || tablename) DESC\n LIMIT 1\n )\n ) AS stats\nUNION ALL\nSELECT \n 'cache_stats',\n jsonb_build_object(\n 'cache_hit_ratio', (\n SELECT round(100.0 * sum(heap_blks_hit) / nullif(sum(heap_blks_hit) + sum(heap_blks_read), 0), 2)\n FROM pg_statio_user_tables\n )\n )\nUNION ALL\nSELECT \n 'connection_stats',\n jsonb_build_object(\n 'active_connections', (SELECT COUNT(*) FROM pg_stat_activity WHERE state = 'active'),\n 'idle_connections', (SELECT COUNT(*) FROM pg_stat_activity WHERE state = 'idle'),\n 'max_connections', current_setting('max_connections')::int\n );\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":25789,"content_sha256":"d1b3f61d0c9aee023982a1b4e122a97475a4a5e46fdfa016ee3ca60a1a4415a0"},{"filename":"references/plpgsql-table-api.md","content":"# PL/pgSQL & Table API Patterns\n\n## Table of Contents\n1. [Table API Philosophy](#table-api-philosophy)\n2. [Schema Organization](#schema-organization)\n3. [Functions vs Procedures](#functions-vs-procedures)\n4. [Function Structure](#function-structure)\n5. [Parameter Conventions](#parameter-conventions)\n6. [Return Types](#return-types)\n7. [Error Handling](#error-handling)\n8. [Trigger Patterns](#trigger-patterns)\n9. [Security Patterns](#security-patterns)\n10. [Performance Attributes](#performance-attributes)\n11. [Complete Examples](#complete-examples)\n\n## Table API Philosophy\n\n### Core Principle\n\nApplications should never access tables directly. All data access flows through a procedural API:\n\n```mermaid\nflowchart LR\n APP[\"🖥️ Application\"] -->|\"CALL/SELECT\"| API[\"api schema\u003cbr/>Functions &\u003cbr/>Procedures\"]\n API -->|\"SECURITY\u003cbr/>DEFINER\"| DATA[(\"data schema\u003cbr/>Tables\")]\n API -->|\"uses\"| PRIV[\"private schema\u003cbr/>Helpers\"]\n PRIV -->|\"triggers\"| DATA\n \n APP x-.-x|\"❌ NO DIRECT\u003cbr/>ACCESS\"| DATA\n \n style API fill:#c8e6c9\n style DATA fill:#fff3e0\n style PRIV fill:#e1bee7\n```\n\n### Benefits\n- **Encapsulation**: Schema changes don't break applications\n- **Security**: Grant EXECUTE on functions, not SELECT/INSERT on tables\n- **Performance**: Logic executes close to data, reducing round-trips\n- **Consistency**: Business rules enforced in one place\n- **Auditability**: Single point for logging all data access\n\n## Schema Organization\n\n### Three-Schema Pattern\n\n```mermaid\nflowchart TB\n subgraph API_SCHEMA[\"api schema - External Interface\"]\n direction LR\n GET[\"get_* functions\u003cbr/>(read single)\"]\n SELECT[\"select_* functions\u003cbr/>(read multiple)\"]\n INSERT[\"insert_* procedures\u003cbr/>(create)\"]\n UPDATE[\"update_* procedures\u003cbr/>(modify)\"]\n DELETE[\"delete_* procedures\u003cbr/>(remove)\"]\n end\n \n subgraph PRIVATE_SCHEMA[\"private schema - Internal Logic\"]\n direction LR\n TRIGGERS[\"Trigger Functions\u003cbr/>set_updated_at()\"]\n HELPERS[\"Helper Functions\u003cbr/>hash_password()\"]\n VALIDATORS[\"Validators\u003cbr/>validate_email()\"]\n end\n \n subgraph DATA_SCHEMA[\"data schema - Tables Only\"]\n direction LR\n T1[(\"customers\")]\n T2[(\"orders\")]\n T3[(\"order_items\")]\n end\n \n API_SCHEMA --> DATA_SCHEMA\n API_SCHEMA --> PRIVATE_SCHEMA\n PRIVATE_SCHEMA --> DATA_SCHEMA\n \n style API_SCHEMA fill:#c8e6c9\n style PRIVATE_SCHEMA fill:#fff3e0\n style DATA_SCHEMA fill:#e3f2fd\n```\n\n```sql\n-- data schema: Tables only (no direct access)\nCREATE SCHEMA data;\n\n-- private schema: Internal functions, triggers\nCREATE SCHEMA private;\n\n-- api schema: External interface (Table API)\nCREATE SCHEMA api;\n```\n\n### What Goes Where\n\n| Schema | Contains | Access |\n|--------|----------|--------|\n| `data` | Tables, indexes, constraints | Internal only |\n| `private` | Triggers, internal helpers, password hashing | Internal only |\n| `api` | Public functions, procedures, views | Applications |\n\n### API Structure\n\n```sql\n-- Read operations: Functions (in api schema)\napi.select_orders_by_customer(in_customer_id)\napi.get_order(in_order_id)\n\n-- Write operations: Procedures (in api schema)\napi.insert_order(in_customer_id, in_items, INOUT io_id)\napi.update_order_status(in_order_id, in_status)\napi.delete_order(in_order_id)\n\n-- Grant access to api schema only\nGRANT USAGE ON SCHEMA api TO app_role;\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA api TO app_role;\nGRANT EXECUTE ON ALL PROCEDURES IN SCHEMA api TO app_role;\n\n-- NO grants on data or private schemas!\n```\n\n## Functions vs Procedures\n\n### Decision Flowchart\n\n```mermaid\nflowchart TD\n START([What do you need?]) --> Q1{Modifies data?}\n \n Q1 -->|No - Read only| Q2{Returns data?}\n Q1 -->|Yes - INSERT/UPDATE/DELETE| PROC[Use PROCEDURE]\n \n Q2 -->|Yes| FUNC[Use FUNCTION]\n Q2 -->|No - Side effect only| PROC\n \n FUNC --> F_ATTRS[\"Add attributes:\u003cbr/>• STABLE or IMMUTABLE\u003cbr/>• SECURITY DEFINER\u003cbr/>• SET search_path\"]\n \n PROC --> P_ATTRS[\"Add attributes:\u003cbr/>• SECURITY DEFINER\u003cbr/>• SET search_path\u003cbr/>• Use INOUT for return values\"]\n \n style FUNC fill:#c8e6c9\n style PROC fill:#bbdefb\n```\n\n### Functions (SELECT operations)\n\n```sql\n-- Use functions for:\n-- - Query operations (SELECT)\n-- - Returning data\n-- - Pure computations\n-- - Can be used in SQL expressions\n\nCREATE FUNCTION api.get_customer_balance(in_customer_id uuid)\nRETURNS numeric\nLANGUAGE sql\nSTABLE -- Doesn't modify data\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT COALESCE(SUM(amount), 0)\n FROM data.transactions\n WHERE customer_id = in_customer_id;\n$;\n\n-- Usage\nSELECT api.get_customer_balance('...');\n```\n\n### Procedures (Mutations)\n\n```sql\n-- Use procedures for:\n-- - INSERT, UPDATE, DELETE operations\n-- - Transaction control (COMMIT, ROLLBACK)\n-- - Multiple DML statements\n-- - Cannot be used in SQL expressions\n\nCREATE PROCEDURE api.transfer_funds(\n in_from_account uuid,\n in_to_account uuid,\n in_amount numeric\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n -- Debit source\n UPDATE data.accounts \n SET balance = balance - in_amount \n WHERE id = in_from_account;\n \n -- Credit destination\n UPDATE data.accounts \n SET balance = balance + in_amount \n WHERE id = in_to_account;\n \n -- Log transfer\n INSERT INTO data.transfers (from_account, to_account, amount)\n VALUES (in_from_account, in_to_account, in_amount);\n \n COMMIT;\nEND;\n$;\n\n-- Usage\nCALL api.transfer_funds('...', '...', 100.00);\n```\n\n## Function Structure\n\n### Standard Template\n\n```sql\nCREATE OR REPLACE FUNCTION api.action_entity_by_filter(\n -- Input parameters (prefix with in_)\n in_param1 type,\n in_param2 type DEFAULT default_value\n)\nRETURNS return_type\nLANGUAGE plpgsql -- or sql for simple queries\nVOLATILITY -- IMMUTABLE, STABLE, or VOLATILE\nPARALLEL SAFETY -- PARALLEL SAFE, PARALLEL RESTRICTED, PARALLEL UNSAFE\nSECURITY DEFINER -- Required for api schema functions\nSET search_path = data, private, pg_temp -- Always set for SECURITY DEFINER\nAS $\nDECLARE\n -- Variable declarations\n l_result type;\nBEGIN\n -- Function body\n\n RETURN l_result;\nEND;\n$;\n\nCOMMENT ON FUNCTION api.action_entity_by_filter(type, type) \n IS 'Brief description of what this function does';\n```\n\n### SQL Language (Preferred for Simple Queries)\n\n```sql\n-- Use LANGUAGE sql when function is a single query\nCREATE FUNCTION api.select_active_customers()\nRETURNS TABLE (id uuid, email text, name text, created_at timestamptz)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name, created_at \n FROM data.customers \n WHERE is_active = true \n ORDER BY name;\n$;\n```\n\n## Parameter Conventions\n\n### Naming\n\n```sql\n-- Always prefix parameters with in_ to avoid column name conflicts\nCREATE FUNCTION api.select_orders_by_customer(\n in_customer_id uuid, -- Required parameter\n in_status text DEFAULT NULL, -- Optional filter\n in_limit integer DEFAULT 100, -- Pagination\n in_offset integer DEFAULT 0\n)\n```\n\n### Parameter Modes\n\n```sql\n-- IN: Input only (default)\n-- OUT: Output only (for procedures returning values)\n-- INOUT: Both input and output\n\nCREATE PROCEDURE api.insert_order(\n in_customer_id uuid, -- IN (input)\n in_subtotal numeric, -- IN (input)\n INOUT io_id uuid DEFAULT NULL -- INOUT (returns generated ID)\n)\nLANGUAGE plpgsql\nAS $\nBEGIN\n INSERT INTO data.orders (customer_id, subtotal)\n VALUES (in_customer_id, in_subtotal)\n RETURNING id INTO io_id;\nEND;\n$;\n\n-- Usage: returns the generated ID\nCALL api.insert_order('customer-uuid', 99.99, NULL);\n```\n\n### Named Parameter Calls\n\n```sql\n-- Always use named parameters for clarity\nSELECT * FROM api.select_orders_by_customer(\n in_customer_id := 'uuid-here',\n in_status := 'pending',\n in_limit := 50\n);\n\nCALL api.insert_order(\n in_customer_id := 'uuid-here',\n in_subtotal := 199.99,\n io_id := NULL\n);\n```\n\n## Return Types\n\n> **Note**: Examples below are simplified for clarity. In production, all `api` schema functions should include `SECURITY DEFINER` and `SET search_path = data, private, pg_temp`. See [Security Patterns](#security-patterns) for complete templates.\n\n### Single Value\n\n```sql\nCREATE FUNCTION api.get_order_total(in_order_id uuid)\nRETURNS numeric\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT total FROM data.orders WHERE id = in_order_id;\n$;\n```\n\n### Single Row (Avoid RETURNS table_type)\n\n```sql\n-- AVOID: Returning entire row type exposes all columns including sensitive ones\n-- CREATE FUNCTION api.get_order(in_id uuid) RETURNS data.orders ...\n\n-- BETTER: Return explicit columns only\nCREATE FUNCTION api.get_order(in_id uuid)\nRETURNS TABLE (id uuid, status text, total numeric, created_at timestamptz)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, status, total, created_at FROM data.orders WHERE id = in_id;\n$;\n```\n\n### Multiple Rows\n\n```sql\n-- Return custom columns (preferred - explicit control)\nCREATE FUNCTION api.select_order_summary(in_customer_id uuid)\nRETURNS TABLE (\n order_id uuid,\n total numeric,\n status text,\n item_count bigint\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT \n o.id,\n o.total,\n o.status,\n COUNT(oi.id)\n FROM data.orders o\n LEFT JOIN data.order_items oi ON oi.order_id = o.id\n WHERE o.customer_id = in_customer_id\n GROUP BY o.id;\n$;\n```\n\n### Void (No Return)\n\n```sql\nCREATE PROCEDURE api.delete_expired_sessions()\nLANGUAGE sql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n DELETE FROM data.sessions WHERE expires_at \u003c now();\n$;\n```\n\n## Error Handling\n\n### Raising Exceptions\n\n```sql\nCREATE PROCEDURE api.update_order_status(\n in_order_id uuid,\n in_new_status text\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_current_status text;\nBEGIN\n -- Get current status\n SELECT status INTO l_current_status\n FROM data.orders\n WHERE id = in_order_id;\n\n -- Check order exists\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Order not found: %', in_order_id\n USING ERRCODE = 'P0002'; -- no_data_found\n END IF;\n\n -- Validate state transition\n IF l_current_status = 'cancelled' THEN\n RAISE EXCEPTION 'Cannot modify cancelled order: %', in_order_id\n USING ERRCODE = 'P0001', -- Custom error code\n HINT = 'Create a new order instead';\n END IF;\n\n -- Perform update\n UPDATE data.orders\n SET status = in_new_status,\n updated_at = now()\n WHERE id = in_order_id;\nEND;\n$;\n```\n\n### Exception Handling\n\n```sql\nCREATE PROCEDURE api.safe_transfer_funds(\n in_from_account uuid,\n in_to_account uuid,\n in_amount numeric\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_from_balance numeric;\nBEGIN\n -- Lock source account\n SELECT balance INTO l_from_balance\n FROM data.accounts\n WHERE id = in_from_account\n FOR UPDATE;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Source account not found';\n END IF;\n\n IF l_from_balance \u003c in_amount THEN\n RAISE EXCEPTION 'Insufficient funds: have %, need %',\n l_from_balance, in_amount;\n END IF;\n\n -- Perform transfer\n UPDATE data.accounts SET balance = balance - in_amount\n WHERE id = in_from_account;\n\n UPDATE data.accounts SET balance = balance + in_amount\n WHERE id = in_to_account;\n\nEXCEPTION\n WHEN foreign_key_violation THEN\n RAISE EXCEPTION 'Destination account not found';\n WHEN OTHERS THEN\n RAISE EXCEPTION 'Transfer failed: %', SQLERRM;\nEND;\n$;\n```\n\n## Trigger Patterns\n\n### Trigger Functions Go in `private` Schema\n\nTrigger functions are internal implementation details and belong in the `private` schema:\n\n```sql\n-- Trigger function in private schema\nCREATE FUNCTION private.set_updated_at()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nBEGIN\n NEW.updated_at := now();\n RETURN NEW;\nEND;\n$;\n\n-- Apply to tables in data schema\nCREATE TRIGGER orders_bu_updated_trg\n BEFORE UPDATE ON data.orders\n FOR EACH ROW\n EXECUTE FUNCTION private.set_updated_at();\n\nCREATE TRIGGER customers_bu_updated_trg\n BEFORE UPDATE ON data.customers\n FOR EACH ROW\n EXECUTE FUNCTION private.set_updated_at();\n```\n\n### Audit Logging Trigger\n\n```sql\n-- Audit trigger in private schema\nCREATE FUNCTION private.log_changes()\nRETURNS trigger\nLANGUAGE plpgsql\nSECURITY DEFINER -- Run as owner to write to audit schema\nSET search_path = app_audit, pg_temp\nAS $\nBEGIN\n IF TG_OP = 'DELETE' THEN\n INSERT INTO app_audit.changelog (\n table_name, operation, old_data, changed_by\n ) VALUES (\n TG_TABLE_NAME, TG_OP, to_jsonb(OLD), current_user\n );\n RETURN OLD;\n ELSE\n INSERT INTO app_audit.changelog (\n table_name, operation, old_data, new_data, changed_by\n ) VALUES (\n TG_TABLE_NAME, \n TG_OP, \n CASE WHEN TG_OP = 'UPDATE' THEN to_jsonb(OLD) END,\n to_jsonb(NEW), \n current_user\n );\n RETURN NEW;\n END IF;\nEND;\n$;\n\nCREATE TRIGGER orders_audit_trg\n AFTER INSERT OR UPDATE OR DELETE ON data.orders\n FOR EACH ROW\n EXECUTE FUNCTION app_audit.log_changes();\n```\n\n### Validation Trigger\n\n```sql\nCREATE FUNCTION api.validate_order_transition()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nDECLARE\n co_valid_transitions jsonb := '{\n \"draft\": [\"pending\", \"cancelled\"],\n \"pending\": [\"confirmed\", \"cancelled\"],\n \"confirmed\": [\"shipped\", \"cancelled\"],\n \"shipped\": [\"delivered\"],\n \"delivered\": [],\n \"cancelled\": []\n }'::jsonb;\nBEGIN\n IF TG_OP = 'UPDATE' AND OLD.status != NEW.status THEN\n IF NOT (co_valid_transitions->OLD.status) ? NEW.status THEN\n RAISE EXCEPTION 'Invalid status transition: % -> %',\n OLD.status, NEW.status;\n END IF;\n END IF;\n RETURN NEW;\nEND;\n$;\n```\n\n## Security Patterns\n\n### API Functions: Always Use SECURITY DEFINER\n\nFor the Table API pattern with schema separation, API functions **must** use `SECURITY DEFINER`:\n\n```sql\n-- API functions need SECURITY DEFINER to access data schema\nCREATE FUNCTION api.select_my_orders(in_customer_id uuid)\nRETURNS TABLE (id uuid, status text, total numeric, created_at timestamptz)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER -- Required: caller has no direct data access\nSET search_path = data, private, pg_temp -- Required with SECURITY DEFINER\nAS $\n SELECT id, status, total, created_at\n FROM data.orders \n WHERE customer_id = in_customer_id;\n$;\n```\n\n### Critical: Always Set search_path\n\n```sql\n-- WRONG: Vulnerable to search_path attacks\nCREATE FUNCTION api.get_customer(in_id uuid)\nRETURNS TABLE (...)\nSECURITY DEFINER\nAS $ ... $; -- Missing SET search_path!\n\n-- CORRECT: Explicit search_path\nCREATE FUNCTION api.get_customer(in_id uuid)\nRETURNS TABLE (id uuid, email text, name text)\nSECURITY DEFINER\nSET search_path = data, private, pg_temp -- Always include pg_temp last\nAS $\n SELECT id, email, name FROM data.customers WHERE id = in_id;\n$;\n```\n\n### Internal Functions: Use SECURITY INVOKER\n\nFunctions in `private` schema that are called by other functions can use INVOKER:\n\n```sql\n-- Internal helper (called by api functions, not directly)\nCREATE FUNCTION private.hash_password(in_password text)\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nSECURITY INVOKER -- Inherits caller's permissions (the api function)\nAS $\n SELECT crypt(in_password, gen_salt('bf', 10));\n$;\n```\n\n### Permission Model\n\n```sql\n-- Grant execute on api schema only\nGRANT USAGE ON SCHEMA api TO app_service;\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA api TO app_service;\nGRANT EXECUTE ON ALL PROCEDURES IN SCHEMA api TO app_service;\n\n-- NO grants on data or private schemas\n-- api functions access them via SECURITY DEFINER\n\n-- Set default search_path for application connections\nALTER ROLE app_service SET search_path = api, pg_temp;\n```\n\n## Performance Attributes\n\n### Volatility\n\n```sql\n-- IMMUTABLE: Same inputs always produce same output, no DB access\nCREATE FUNCTION api.calculate_tax(in_amount numeric, in_rate numeric)\nRETURNS numeric\nLANGUAGE sql\nIMMUTABLE -- Can be used in indexes, optimized heavily\nPARALLEL SAFE\nAS $\n SELECT in_amount * in_rate;\n$;\n\n-- STABLE: Same inputs produce same output within single query\n-- Can read DB but results don't change during query\nCREATE FUNCTION api.get_current_rate(in_product_id uuid)\nRETURNS numeric\nLANGUAGE sql\nSTABLE -- Safe for use in WHERE clauses\nPARALLEL SAFE\nAS $\n SELECT rate FROM data.rates WHERE product_id = in_product_id;\n$;\n\n-- VOLATILE: May return different results on each call (default)\nCREATE FUNCTION api.get_next_order_number()\nRETURNS bigint\nLANGUAGE sql\nVOLATILE -- Uses sequence, changes state\nAS $\n SELECT nextval('data.order_number_seq');\n$;\n```\n\n### Parallel Safety\n\n```sql\n-- PARALLEL SAFE: Can run in parallel workers\n-- PARALLEL RESTRICTED: Must run in leader (references parallel-unsafe state)\n-- PARALLEL UNSAFE: Forces entire query to run sequentially\n\nCREATE FUNCTION api.expensive_calculation(in_data jsonb)\nRETURNS jsonb\nLANGUAGE plpgsql\nSTABLE\nPARALLEL SAFE -- Enable parallel execution when in parallel query\nAS $\n-- ...\n$;\n```\n\n### Other Attributes\n\n```sql\nCREATE FUNCTION api.validate_email(in_email text)\nRETURNS boolean\nLANGUAGE sql\nIMMUTABLE\nPARALLEL SAFE\nRETURNS NULL ON NULL INPUT -- Skip function if any arg is NULL\nLEAKPROOF -- Doesn't reveal info about inputs (for security)\nCOST 10 -- Planner hint: relative execution cost\nAS $\n SELECT in_email ~* '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z]{2,}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.

;\n$;\n```\n\n## Complete Examples\n\n### Full CRUD Table API\n\n```sql\n-- ============================================\n-- Customer Table API (in api schema)\n-- ============================================\n\n-- SELECT: Get by ID (excludes sensitive fields)\nCREATE FUNCTION api.get_customer(in_id uuid)\nRETURNS TABLE (\n id uuid, email text, name text, \n is_active boolean, created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name, is_active, created_at\n FROM data.customers \n WHERE id = in_id;\n$;\n\n-- SELECT: Get by email\nCREATE FUNCTION api.get_customer_by_email(in_email text)\nRETURNS TABLE (\n id uuid, email text, name text,\n is_active boolean, created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name, is_active, created_at\n FROM data.customers \n WHERE lower(email) = lower(in_email);\n$;\n\n-- SELECT: Search with pagination\nCREATE FUNCTION api.select_customers(\n in_search text DEFAULT NULL,\n in_is_active boolean DEFAULT NULL,\n in_limit integer DEFAULT 100,\n in_offset integer DEFAULT 0\n)\nRETURNS TABLE (\n id uuid, email text, name text,\n is_active boolean, created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name, is_active, created_at\n FROM data.customers\n WHERE (in_search IS NULL OR name ILIKE '%' || in_search || '%')\n AND (in_is_active IS NULL OR is_active = in_is_active)\n ORDER BY name\n LIMIT in_limit\n OFFSET in_offset;\n$;\n\n-- INSERT\nCREATE PROCEDURE api.insert_customer(\n in_email text,\n in_name text,\n in_password text,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.customers (email, name, password_hash)\n VALUES (lower(trim(in_email)), trim(in_name), private.hash_password(in_password))\n RETURNING id INTO io_id;\nEND;\n$;\n\n-- UPDATE\nCREATE PROCEDURE api.update_customer(\n in_id uuid,\n in_name text DEFAULT NULL,\n in_email text DEFAULT NULL,\n in_is_active boolean DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n UPDATE data.customers\n SET name = COALESCE(trim(in_name), name),\n email = COALESCE(lower(trim(in_email)), email),\n is_active = COALESCE(in_is_active, is_active)\n WHERE id = in_id;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Customer not found: %', in_id\n USING ERRCODE = 'P0002';\n END IF;\nEND;\n$;\n\n-- DELETE (soft delete)\nCREATE PROCEDURE api.delete_customer(in_id uuid)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n UPDATE data.customers\n SET is_active = false,\n deleted_at = now()\n WHERE id = in_id;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Customer not found: %', in_id\n USING ERRCODE = 'P0002';\n END IF;\nEND;\n$;\n\n-- UPSERT\nCREATE PROCEDURE api.upsert_customer(\n in_email text,\n in_name text,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.customers (email, name)\n VALUES (lower(trim(in_email)), trim(in_name))\n ON CONFLICT (lower(email)) DO UPDATE\n SET name = EXCLUDED.name\n RETURNING id INTO io_id;\nEND;\n$;\n```\n\n### Using PostgreSQL 18 RETURNING with OLD/NEW\n\n```sql\n-- Track changes using PG18 OLD/NEW in RETURNING\nCREATE FUNCTION api.update_order_with_history(\n in_order_id uuid,\n in_new_status text\n)\nRETURNS TABLE (\n order_id uuid,\n old_status text,\n new_status text,\n changed_at timestamptz\n)\nLANGUAGE sql\nAS $\n UPDATE data.orders\n SET status = in_new_status,\n updated_at = now()\n WHERE id = in_order_id\n RETURNING \n id,\n OLD.status,\n NEW.status,\n NEW.updated_at;\n$;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":22001,"content_sha256":"bc5300b64231d5b51912ffd00e75231adb4f6b921401fe0983ff0c4c6323e43d"},{"filename":"references/postgis-patterns.md","content":"# PostGIS Spatial Patterns\n\nThis document covers PostGIS integration for geographic data including spatial types, indexing, common queries, and performance optimization.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Installation & Setup](#installation--setup)\n3. [Spatial Data Types](#spatial-data-types)\n4. [Schema Design](#schema-design)\n5. [Spatial Indexing](#spatial-indexing)\n6. [Query Patterns](#query-patterns)\n7. [Performance Optimization](#performance-optimization)\n8. [Common Use Cases](#common-use-cases)\n\n## Overview\n\n### When to Use PostGIS\n\n| Use Case | PostGIS | Alternative |\n|----------|---------|-------------|\n| Store locations | ✅ Native | Point columns work |\n| Distance queries | ✅ Optimized | Can use formulas |\n| Polygon/area operations | ✅ Required | Not feasible |\n| Route planning | ⚠️ Limited | Dedicated routing |\n| Map rendering | ⚠️ Data storage | Map servers |\n\n### Core Concepts\n\n```mermaid\nflowchart LR\n subgraph TYPES[\"Geometry Types\"]\n POINT[\"Point\"]\n LINE[\"LineString\"]\n POLYGON[\"Polygon\"]\n MULTI[\"Multi*\"]\n end\n\n subgraph SRID[\"Coordinate Systems\"]\n WGS84[\"SRID 4326\u003cbr/>WGS84 (GPS)\"]\n WEBMERC[\"SRID 3857\u003cbr/>Web Mercator\"]\n LOCAL[\"Local projections\"]\n end\n\n subgraph OPS[\"Operations\"]\n DISTANCE[\"ST_Distance\"]\n WITHIN[\"ST_Within\"]\n INTERSECTS[\"ST_Intersects\"]\n BUFFER[\"ST_Buffer\"]\n end\n\n TYPES --> SRID\n SRID --> OPS\n```\n\n## Installation & Setup\n\n### Install PostGIS\n\n```sql\n-- Install PostGIS extension\nCREATE EXTENSION IF NOT EXISTS postgis;\n\n-- Optional: Additional extensions\nCREATE EXTENSION IF NOT EXISTS postgis_topology;\nCREATE EXTENSION IF NOT EXISTS postgis_raster;\nCREATE EXTENSION IF NOT EXISTS fuzzystrmatch; -- For geocoding\nCREATE EXTENSION IF NOT EXISTS postgis_tiger_geocoder;\n\n-- Verify installation\nSELECT PostGIS_Version();\nSELECT PostGIS_Full_Version();\n```\n\n### Spatial Reference Systems\n\n```sql\n-- Common SRIDs\n-- 4326: WGS84 (GPS coordinates, lat/lon in degrees)\n-- 3857: Web Mercator (Google Maps, units in meters)\n-- Local: State Plane, UTM zones\n\n-- View available SRIDs\nSELECT srid, auth_name, auth_srid, srtext\nFROM spatial_ref_sys\nWHERE auth_name = 'EPSG'\nLIMIT 10;\n\n-- Add custom SRID if needed\nINSERT INTO spatial_ref_sys (srid, auth_name, auth_srid, srtext, proj4text)\nVALUES (900913, 'EPSG', 900913, 'WGS 84 / Pseudo-Mercator', '+proj=merc...');\n```\n\n## Spatial Data Types\n\n### Geometry vs Geography\n\n```sql\n-- Geometry: Planar coordinates, faster, use for local areas\n-- Geography: Spherical coordinates, accurate for global data\n\n-- Geometry column (flat earth math)\nCREATE TABLE data.buildings (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n location geometry(Point, 4326) -- 2D point in WGS84\n);\n\n-- Geography column (spherical earth math)\nCREATE TABLE data.cities (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n location geography(Point, 4326) -- Accurate global distances\n);\n\n-- Comparison\n-- Geometry: ST_Distance returns units of SRID (degrees for 4326)\n-- Geography: ST_Distance returns meters (always accurate)\n```\n\n### Geometry Types\n\n```sql\n-- Point\nSELECT ST_GeomFromText('POINT(-122.4194 37.7749)', 4326);\nSELECT ST_MakePoint(-122.4194, 37.7749); -- Shorthand\n\n-- LineString\nSELECT ST_GeomFromText('LINESTRING(-122.4 37.7, -122.5 37.8, -122.6 37.9)', 4326);\n\n-- Polygon\nSELECT ST_GeomFromText('POLYGON((-122.4 37.7, -122.5 37.7, -122.5 37.8, -122.4 37.8, -122.4 37.7))', 4326);\n\n-- MultiPoint, MultiLineString, MultiPolygon\nSELECT ST_GeomFromText('MULTIPOINT((-122.4 37.7), (-122.5 37.8))', 4326);\n\n-- GeometryCollection\nSELECT ST_GeomFromText('GEOMETRYCOLLECTION(POINT(-122.4 37.7), LINESTRING(-122.4 37.7, -122.5 37.8))', 4326);\n```\n\n## Schema Design\n\n### Location Table\n\n```sql\nCREATE TABLE data.locations (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n address text,\n\n -- Geographic point (accurate global distances)\n location geography(Point, 4326) NOT NULL,\n\n -- Derived columns for convenience\n latitude double precision GENERATED ALWAYS AS (ST_Y(location::geometry)) STORED,\n longitude double precision GENERATED ALWAYS AS (ST_X(location::geometry)) STORED,\n\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Spatial index\nCREATE INDEX locations_location_idx ON data.locations USING gist (location);\n\n-- B-tree indexes for sorting by lat/lon\nCREATE INDEX locations_lat_idx ON data.locations (latitude);\nCREATE INDEX locations_lon_idx ON data.locations (longitude);\n```\n\n### Areas/Regions Table\n\n```sql\nCREATE TABLE data.regions (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n region_type text NOT NULL, -- 'city', 'state', 'country', 'custom'\n\n -- Polygon boundary\n boundary geography(Polygon, 4326) NOT NULL,\n\n -- Cached calculations\n area_sq_km double precision GENERATED ALWAYS AS (\n ST_Area(boundary::geography) / 1000000\n ) STORED,\n\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE INDEX regions_boundary_idx ON data.regions USING gist (boundary);\n```\n\n### Points of Interest\n\n```sql\nCREATE TABLE data.pois (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n category text NOT NULL,\n location geography(Point, 4326) NOT NULL,\n metadata jsonb NOT NULL DEFAULT '{}',\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE INDEX pois_location_idx ON data.pois USING gist (location);\nCREATE INDEX pois_category_idx ON data.pois (category);\n\n-- Composite spatial + category index\nCREATE INDEX pois_category_location_idx ON data.pois USING gist (location)\n WHERE category IS NOT NULL;\n```\n\n## Spatial Indexing\n\n### GiST Index (General)\n\n```sql\n-- GiST: Default spatial index, good for most use cases\nCREATE INDEX locations_geom_idx ON data.locations USING gist (location);\n\n-- Supports: &&, @, ~, \u003c->, ST_Intersects, ST_Contains, ST_Within, ST_DWithin\n```\n\n### SP-GiST Index (Space-Partitioned)\n\n```sql\n-- SP-GiST: Better for highly clustered point data\nCREATE INDEX locations_spgist_idx ON data.locations USING spgist (location);\n\n-- Good for quad-tree type queries\n```\n\n### BRIN Index (Block Range)\n\n```sql\n-- BRIN: For very large tables with spatial locality\n-- Much smaller than GiST, but less precise\nCREATE INDEX locations_brin_idx ON data.locations USING brin (location)\n WITH (pages_per_range = 128);\n\n-- Best when data is physically ordered by location\n```\n\n### Index Selection Guide\n\n| Scenario | Index Type | Notes |\n|----------|------------|-------|\n| General spatial queries | GiST | Default choice |\n| Clustered points | SP-GiST | Quad-tree efficient |\n| Very large, ordered data | BRIN | Smallest size |\n| K-nearest neighbor | GiST | Only GiST supports \u003c-> |\n\n## Query Patterns\n\n### Distance Queries\n\n```sql\n-- Find locations within distance (geography - meters)\nSELECT id, name, ST_Distance(location, ST_MakePoint(-122.4194, 37.7749)::geography) AS distance_m\nFROM data.locations\nWHERE ST_DWithin(location, ST_MakePoint(-122.4194, 37.7749)::geography, 5000) -- 5km\nORDER BY distance_m;\n\n-- K-Nearest Neighbors (KNN)\nSELECT id, name, location \u003c-> ST_MakePoint(-122.4194, 37.7749)::geography AS distance\nFROM data.locations\nORDER BY location \u003c-> ST_MakePoint(-122.4194, 37.7749)::geography\nLIMIT 10;\n```\n\n### Containment Queries\n\n```sql\n-- Find all POIs within a region\nSELECT p.id, p.name, p.category\nFROM data.pois p\nJOIN data.regions r ON ST_Within(p.location::geometry, r.boundary::geometry)\nWHERE r.name = 'San Francisco';\n\n-- Find which region contains a point\nSELECT id, name\nFROM data.regions\nWHERE ST_Contains(boundary::geometry, ST_MakePoint(-122.4194, 37.7749)::geometry);\n```\n\n### Intersection Queries\n\n```sql\n-- Find regions that intersect with a bounding box\nSELECT id, name\nFROM data.regions\nWHERE ST_Intersects(\n boundary::geometry,\n ST_MakeEnvelope(-122.5, 37.7, -122.3, 37.9, 4326)\n);\n\n-- Find overlapping regions\nSELECT a.name AS region_a, b.name AS region_b\nFROM data.regions a\nJOIN data.regions b ON ST_Intersects(a.boundary::geometry, b.boundary::geometry)\nWHERE a.id \u003c b.id; -- Avoid duplicates\n```\n\n### API Functions\n\n```sql\n-- Find nearby locations\nCREATE FUNCTION api.find_nearby_locations(\n in_lat double precision,\n in_lon double precision,\n in_radius_meters integer DEFAULT 1000,\n in_limit integer DEFAULT 20\n)\nRETURNS TABLE (\n id uuid,\n name text,\n distance_meters double precision,\n latitude double precision,\n longitude double precision\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT\n id,\n name,\n ST_Distance(location, ST_MakePoint(in_lon, in_lat)::geography) AS distance_meters,\n ST_Y(location::geometry) AS latitude,\n ST_X(location::geometry) AS longitude\n FROM data.locations\n WHERE ST_DWithin(location, ST_MakePoint(in_lon, in_lat)::geography, in_radius_meters)\n ORDER BY location \u003c-> ST_MakePoint(in_lon, in_lat)::geography\n LIMIT in_limit;\n$;\n\n-- Find POIs by category near a point\nCREATE FUNCTION api.find_pois_nearby(\n in_lat double precision,\n in_lon double precision,\n in_categories text[],\n in_radius_meters integer DEFAULT 500\n)\nRETURNS TABLE (\n id uuid,\n name text,\n category text,\n distance_meters double precision\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT\n id,\n name,\n category,\n ST_Distance(location, ST_MakePoint(in_lon, in_lat)::geography) AS distance_meters\n FROM data.pois\n WHERE ST_DWithin(location, ST_MakePoint(in_lon, in_lat)::geography, in_radius_meters)\n AND (in_categories IS NULL OR category = ANY(in_categories))\n ORDER BY location \u003c-> ST_MakePoint(in_lon, in_lat)::geography;\n$;\n```\n\n## Performance Optimization\n\n### Query Optimization\n\n```sql\n-- 1. Use ST_DWithin instead of ST_Distance \u003c X\n-- ❌ Bad: Calculates distance for all rows\nSELECT * FROM data.locations\nWHERE ST_Distance(location, ref_point) \u003c 5000;\n\n-- ✅ Good: Uses spatial index\nSELECT * FROM data.locations\nWHERE ST_DWithin(location, ref_point, 5000);\n\n-- 2. Use && (bounding box) for pre-filtering\nSELECT * FROM data.regions\nWHERE boundary && ST_MakeEnvelope(-122.5, 37.7, -122.3, 37.9, 4326)\n AND ST_Intersects(boundary::geometry, query_polygon);\n\n-- 3. Avoid coordinate system conversions in WHERE clause\n-- ❌ Bad: Converts every row\nSELECT * FROM data.locations\nWHERE ST_Distance(ST_Transform(location::geometry, 3857), ref_point_3857) \u003c 5000;\n\n-- ✅ Good: Store in target SRID or convert reference point\nSELECT * FROM data.locations\nWHERE ST_DWithin(location, ST_Transform(ref_point_3857, 4326)::geography, 5000);\n```\n\n### Clustering\n\n```sql\n-- Cluster table by spatial index (physical reordering)\nCLUSTER data.locations USING locations_location_idx;\n\n-- Or use geohash for ordering\nALTER TABLE data.locations ADD COLUMN geohash text\n GENERATED ALWAYS AS (ST_GeoHash(location::geometry, 8)) STORED;\n\nCREATE INDEX locations_geohash_idx ON data.locations (geohash);\n```\n\n### Simplification for Large Polygons\n\n```sql\n-- Simplify complex geometries for faster queries\nUPDATE data.regions\nSET boundary_simplified = ST_Simplify(boundary::geometry, 0.001)::geography;\n\n-- Use simplified version for display, original for precise queries\n```\n\n## Common Use Cases\n\n### Geofencing\n\n```sql\n-- Check if a point is within a geofence\nCREATE FUNCTION api.check_geofence(\n in_device_id uuid,\n in_lat double precision,\n in_lon double precision\n)\nRETURNS TABLE (\n geofence_id uuid,\n geofence_name text,\n is_inside boolean\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT\n g.id AS geofence_id,\n g.name AS geofence_name,\n ST_Within(\n ST_MakePoint(in_lon, in_lat)::geometry,\n g.boundary::geometry\n ) AS is_inside\n FROM data.geofences g\n WHERE g.device_id = in_device_id\n OR g.is_global = true;\n$;\n```\n\n### Store Locator\n\n```sql\nCREATE FUNCTION api.find_stores(\n in_lat double precision,\n in_lon double precision,\n in_max_results integer DEFAULT 10\n)\nRETURNS TABLE (\n store_id uuid,\n store_name text,\n address text,\n distance_km double precision,\n latitude double precision,\n longitude double precision\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT\n s.id,\n s.name,\n s.address,\n round((ST_Distance(s.location, ST_MakePoint(in_lon, in_lat)::geography) / 1000)::numeric, 2) AS distance_km,\n ST_Y(s.location::geometry),\n ST_X(s.location::geometry)\n FROM data.stores s\n WHERE s.is_active = true\n ORDER BY s.location \u003c-> ST_MakePoint(in_lon, in_lat)::geography\n LIMIT in_max_results;\n$;\n```\n\n### Coverage Analysis\n\n```sql\n-- Calculate coverage area for service regions\nSELECT\n r.name,\n ST_Area(r.boundary::geography) / 1000000 AS area_sq_km,\n (SELECT count(*) FROM data.customers c\n WHERE ST_Within(c.location::geometry, r.boundary::geometry)) AS customers_in_region\nFROM data.service_regions r;\n\n-- Find uncovered areas\nSELECT ST_AsGeoJSON(\n ST_Difference(\n (SELECT ST_Union(boundary::geometry) FROM data.target_area),\n (SELECT ST_Union(boundary::geometry) FROM data.service_regions)\n )\n) AS uncovered_geojson;\n```\n\n### Route/Path Storage\n\n```sql\nCREATE TABLE data.routes (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n path geography(LineString, 4326) NOT NULL,\n length_km double precision GENERATED ALWAYS AS (\n ST_Length(path) / 1000\n ) STORED,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Find routes passing through a point\nSELECT id, name, length_km\nFROM data.routes\nWHERE ST_DWithin(path, ST_MakePoint(-122.4, 37.7)::geography, 100); -- 100m buffer\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":14170,"content_sha256":"d14610e37b607f44f86cff3db7da3d72792342a3abfd942b18c628b7f992b6fb"},{"filename":"references/queue-patterns.md","content":"# Database as Queue Patterns\n\nThis document covers using PostgreSQL as a job queue including SKIP LOCKED patterns, LISTEN/NOTIFY, competing consumers, and when to use dedicated queue systems.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Basic Queue Table](#basic-queue-table)\n3. [SKIP LOCKED Pattern](#skip-locked-pattern)\n4. [LISTEN/NOTIFY](#listennotify)\n5. [Competing Consumers](#competing-consumers)\n6. [Retry and Dead Letter](#retry-and-dead-letter)\n7. [Scheduled Jobs](#scheduled-jobs)\n8. [When to Use Dedicated Queues](#when-to-use-dedicated-queues)\n\n## Overview\n\n### PostgreSQL Queue vs Dedicated Queue\n\n| Feature | PostgreSQL Queue | Redis/RabbitMQ/SQS |\n|---------|-----------------|-------------------|\n| Transactional | ✅ Same transaction | ❌ Separate system |\n| Setup complexity | ✅ Already have PG | ❌ Another service |\n| Throughput | ⚠️ Moderate | ✅ High |\n| Durability | ✅ Built-in | ⚠️ Configurable |\n| Exactly-once | ✅ With SKIP LOCKED | ⚠️ At-least-once |\n| Pub/Sub | ⚠️ LISTEN/NOTIFY | ✅ Native |\n\n### When PostgreSQL Queue is Good\n\n```mermaid\nflowchart TD\n START([Need a queue?]) --> Q1{Job volume?}\n\n Q1 -->|\"\u003c 1000/sec\"| Q2{Same DB transaction?}\n Q1 -->|\"> 1000/sec\"| DEDICATED[\"Use dedicated queue\u003cbr/>(Redis, RabbitMQ, SQS)\"]\n\n Q2 -->|\"Yes, critical\"| PG_QUEUE[\"PostgreSQL queue\u003cbr/>(transactional guarantee)\"]\n Q2 -->|\"No\"| Q3{Already have PG?}\n\n Q3 -->|\"Yes\"| PG_QUEUE\n Q3 -->|\"Adding queue system easy\"| DEDICATED\n\n style PG_QUEUE fill:#c8e6c9\n style DEDICATED fill:#bbdefb\n```\n\n## Basic Queue Table\n\n### Queue Table Design\n\n```sql\nCREATE TABLE data.job_queue (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n queue_name text NOT NULL DEFAULT 'default',\n job_type text NOT NULL,\n payload jsonb NOT NULL,\n status text NOT NULL DEFAULT 'pending',\n priority integer NOT NULL DEFAULT 0,\n\n -- Scheduling\n scheduled_at timestamptz NOT NULL DEFAULT now(),\n started_at timestamptz,\n completed_at timestamptz,\n\n -- Processing\n worker_id text,\n attempts integer NOT NULL DEFAULT 0,\n max_attempts integer NOT NULL DEFAULT 3,\n last_error text,\n\n -- Timestamps\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now(),\n\n -- Constraints\n CONSTRAINT job_queue_status_check CHECK (\n status IN ('pending', 'processing', 'completed', 'failed', 'dead')\n )\n);\n\n-- Indexes for queue operations\nCREATE INDEX job_queue_pending_idx ON data.job_queue (queue_name, scheduled_at, priority DESC)\n WHERE status = 'pending';\n\nCREATE INDEX job_queue_processing_idx ON data.job_queue (worker_id, started_at)\n WHERE status = 'processing';\n\nCREATE INDEX job_queue_failed_idx ON data.job_queue (scheduled_at)\n WHERE status = 'failed' AND attempts \u003c max_attempts;\n```\n\n### Enqueue Job\n\n```sql\nCREATE PROCEDURE api.enqueue_job(\n in_job_type text,\n in_payload jsonb,\n in_queue_name text DEFAULT 'default',\n in_priority integer DEFAULT 0,\n in_scheduled_at timestamptz DEFAULT now(),\n INOUT io_job_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.job_queue (job_type, payload, queue_name, priority, scheduled_at)\n VALUES (in_job_type, in_payload, in_queue_name, in_priority, in_scheduled_at)\n RETURNING id INTO io_job_id;\n\n -- Notify listeners\n PERFORM pg_notify('job_queue_' || in_queue_name, io_job_id::text);\nEND;\n$;\n```\n\n## SKIP LOCKED Pattern\n\n### Fetch and Lock Job\n\n```sql\nCREATE FUNCTION api.fetch_job(\n in_queue_name text DEFAULT 'default',\n in_worker_id text DEFAULT NULL\n)\nRETURNS TABLE (\n job_id uuid,\n job_type text,\n payload jsonb,\n attempts integer\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_worker_id text := COALESCE(in_worker_id, 'worker_' || pg_backend_pid());\nBEGIN\n RETURN QUERY\n WITH claimed AS (\n SELECT j.id\n FROM data.job_queue j\n WHERE j.queue_name = in_queue_name\n AND j.status = 'pending'\n AND j.scheduled_at \u003c= now()\n ORDER BY j.priority DESC, j.scheduled_at\n LIMIT 1\n FOR UPDATE SKIP LOCKED\n )\n UPDATE data.job_queue\n SET status = 'processing',\n worker_id = l_worker_id,\n started_at = now(),\n attempts = attempts + 1,\n updated_at = now()\n FROM claimed\n WHERE job_queue.id = claimed.id\n RETURNING job_queue.id, job_queue.job_type, job_queue.payload, job_queue.attempts;\nEND;\n$;\n```\n\n### Complete Job\n\n```sql\nCREATE PROCEDURE api.complete_job(\n in_job_id uuid,\n in_result jsonb DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n UPDATE data.job_queue\n SET status = 'completed',\n completed_at = now(),\n updated_at = now()\n WHERE id = in_job_id\n AND status = 'processing';\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Job not found or not processing: %', in_job_id\n USING ERRCODE = 'P0002';\n END IF;\nEND;\n$;\n```\n\n### Fail Job\n\n```sql\nCREATE PROCEDURE api.fail_job(\n in_job_id uuid,\n in_error text,\n in_retry boolean DEFAULT true\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_job record;\nBEGIN\n SELECT * INTO l_job\n FROM data.job_queue\n WHERE id = in_job_id\n AND status = 'processing'\n FOR UPDATE;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Job not found or not processing: %', in_job_id\n USING ERRCODE = 'P0002';\n END IF;\n\n IF in_retry AND l_job.attempts \u003c l_job.max_attempts THEN\n -- Retry with exponential backoff\n UPDATE data.job_queue\n SET status = 'pending',\n worker_id = NULL,\n started_at = NULL,\n last_error = in_error,\n scheduled_at = now() + (power(2, l_job.attempts) || ' minutes')::interval,\n updated_at = now()\n WHERE id = in_job_id;\n ELSE\n -- Mark as dead (no more retries)\n UPDATE data.job_queue\n SET status = 'dead',\n last_error = in_error,\n completed_at = now(),\n updated_at = now()\n WHERE id = in_job_id;\n END IF;\nEND;\n$;\n```\n\n### Batch Fetch\n\n```sql\nCREATE FUNCTION api.fetch_jobs(\n in_queue_name text DEFAULT 'default',\n in_worker_id text DEFAULT NULL,\n in_batch_size integer DEFAULT 10\n)\nRETURNS TABLE (\n job_id uuid,\n job_type text,\n payload jsonb\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_worker_id text := COALESCE(in_worker_id, 'worker_' || pg_backend_pid());\nBEGIN\n RETURN QUERY\n WITH claimed AS (\n SELECT j.id\n FROM data.job_queue j\n WHERE j.queue_name = in_queue_name\n AND j.status = 'pending'\n AND j.scheduled_at \u003c= now()\n ORDER BY j.priority DESC, j.scheduled_at\n LIMIT in_batch_size\n FOR UPDATE SKIP LOCKED\n )\n UPDATE data.job_queue\n SET status = 'processing',\n worker_id = l_worker_id,\n started_at = now(),\n attempts = attempts + 1,\n updated_at = now()\n FROM claimed\n WHERE job_queue.id = claimed.id\n RETURNING job_queue.id, job_queue.job_type, job_queue.payload;\nEND;\n$;\n```\n\n## LISTEN/NOTIFY\n\n### Real-Time Notifications\n\n```sql\n-- Worker listens for new jobs\nLISTEN job_queue_default;\nLISTEN job_queue_high_priority;\n\n-- Check for notifications (in application)\n-- SELECT * FROM pg_notification_queue();\n\n-- Producer notifies on enqueue (already in enqueue_job)\n-- PERFORM pg_notify('job_queue_' || queue_name, job_id::text);\n```\n\n### Notification Trigger\n\n```sql\nCREATE FUNCTION private.job_queue_notify()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nBEGIN\n IF NEW.status = 'pending' THEN\n PERFORM pg_notify('job_queue_' || NEW.queue_name, NEW.id::text);\n END IF;\n RETURN NEW;\nEND;\n$;\n\nCREATE TRIGGER job_queue_notify_trg\n AFTER INSERT OR UPDATE OF status ON data.job_queue\n FOR EACH ROW\n EXECUTE FUNCTION private.job_queue_notify();\n```\n\n### Application Pattern (Python Example)\n\n```python\nimport psycopg\nimport select\n\nconn = psycopg.connect(\"postgresql://...\")\nconn.execute(\"LISTEN job_queue_default\")\n\nwhile True:\n # Wait for notification or timeout\n if select.select([conn.fileno()], [], [], 5.0) != ([], [], []):\n conn.poll()\n while conn.notifies:\n notify = conn.notifies.pop()\n print(f\"New job: {notify.payload}\")\n # Process job\n process_job(notify.payload)\n else:\n # Timeout: poll for any missed jobs\n poll_for_jobs()\n```\n\n## Competing Consumers\n\n### Multiple Workers\n\n```sql\n-- Each worker has unique ID\n-- SKIP LOCKED ensures each job is processed by one worker only\n\n-- Worker 1\nSELECT * FROM api.fetch_job('default', 'worker-1');\n\n-- Worker 2 (concurrent)\nSELECT * FROM api.fetch_job('default', 'worker-2');\n-- Gets different job (or nothing if no pending jobs)\n```\n\n### Worker Heartbeat\n\n```sql\n-- Add heartbeat column\nALTER TABLE data.job_queue ADD COLUMN heartbeat_at timestamptz;\n\n-- Worker updates heartbeat during processing\nCREATE PROCEDURE api.heartbeat_job(in_job_id uuid)\nLANGUAGE sql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n UPDATE data.job_queue\n SET heartbeat_at = now()\n WHERE id = in_job_id AND status = 'processing';\n$;\n\n-- Recover stuck jobs (worker died)\nCREATE PROCEDURE private.recover_stuck_jobs(in_timeout interval DEFAULT '5 minutes')\nLANGUAGE plpgsql\nAS $\nBEGIN\n UPDATE data.job_queue\n SET status = 'pending',\n worker_id = NULL,\n started_at = NULL,\n updated_at = now()\n WHERE status = 'processing'\n AND (heartbeat_at IS NULL OR heartbeat_at \u003c now() - in_timeout)\n AND started_at \u003c now() - in_timeout;\n\n RAISE NOTICE 'Recovered % stuck jobs', (SELECT count(*) FROM data.job_queue WHERE status = 'pending');\nEND;\n$;\n\n-- Schedule recovery check\nSELECT cron.schedule('recover-stuck-jobs', '*/5 * * * *',\n $CALL private.recover_stuck_jobs()$);\n```\n\n## Retry and Dead Letter\n\n### Retry Configuration\n\n```sql\n-- Per-job retry settings\nCREATE TABLE data.job_retry_config (\n job_type text PRIMARY KEY,\n max_attempts integer NOT NULL DEFAULT 3,\n backoff_base interval NOT NULL DEFAULT '1 minute',\n backoff_factor real NOT NULL DEFAULT 2.0,\n max_backoff interval NOT NULL DEFAULT '1 hour'\n);\n\n-- Default configurations\nINSERT INTO data.job_retry_config (job_type, max_attempts, backoff_base)\nVALUES\n ('email_send', 5, '30 seconds'),\n ('webhook_call', 3, '1 minute'),\n ('report_generate', 2, '5 minutes');\n```\n\n### Calculate Retry Delay\n\n```sql\nCREATE FUNCTION private.calculate_retry_delay(\n in_job_type text,\n in_attempt integer\n)\nRETURNS interval\nLANGUAGE sql\nSTABLE\nAS $\n SELECT LEAST(\n c.backoff_base * power(c.backoff_factor, in_attempt - 1),\n c.max_backoff\n )\n FROM data.job_retry_config c\n WHERE c.job_type = in_job_type\n UNION ALL\n SELECT '1 minute'::interval * power(2, in_attempt - 1) -- Default\n LIMIT 1;\n$;\n```\n\n### Dead Letter Queue\n\n```sql\n-- Separate table for dead jobs (keeps main queue lean)\nCREATE TABLE data.dead_letter_queue (\n id uuid PRIMARY KEY,\n queue_name text NOT NULL,\n job_type text NOT NULL,\n payload jsonb NOT NULL,\n attempts integer NOT NULL,\n last_error text,\n failed_at timestamptz NOT NULL DEFAULT now(),\n original_created_at timestamptz NOT NULL\n);\n\n-- Move to dead letter\nCREATE PROCEDURE private.move_to_dead_letter(in_job_id uuid)\nLANGUAGE plpgsql\nAS $\nBEGIN\n INSERT INTO data.dead_letter_queue (id, queue_name, job_type, payload, attempts, last_error, original_created_at)\n SELECT id, queue_name, job_type, payload, attempts, last_error, created_at\n FROM data.job_queue\n WHERE id = in_job_id;\n\n DELETE FROM data.job_queue WHERE id = in_job_id;\nEND;\n$;\n\n-- Replay dead letter job\nCREATE PROCEDURE api.replay_dead_letter(in_job_id uuid)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.job_queue (id, queue_name, job_type, payload, attempts)\n SELECT id, queue_name, job_type, payload, 0\n FROM data.dead_letter_queue\n WHERE id = in_job_id;\n\n DELETE FROM data.dead_letter_queue WHERE id = in_job_id;\nEND;\n$;\n```\n\n## Scheduled Jobs\n\n### Cron-Style Scheduling\n\n```sql\nCREATE TABLE data.scheduled_jobs (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL UNIQUE,\n job_type text NOT NULL,\n payload jsonb NOT NULL DEFAULT '{}',\n cron_expression text NOT NULL, -- '0 * * * *' = every hour\n queue_name text NOT NULL DEFAULT 'default',\n is_active boolean NOT NULL DEFAULT true,\n last_run_at timestamptz,\n next_run_at timestamptz NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE INDEX scheduled_jobs_next_run_idx ON data.scheduled_jobs (next_run_at)\n WHERE is_active = true;\n```\n\n### Schedule Executor\n\n```sql\n-- Run scheduled jobs that are due\nCREATE PROCEDURE private.run_scheduled_jobs()\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_job record;\n l_job_id uuid;\nBEGIN\n FOR l_job IN\n SELECT *\n FROM data.scheduled_jobs\n WHERE is_active = true\n AND next_run_at \u003c= now()\n FOR UPDATE SKIP LOCKED\n LOOP\n -- Enqueue the job\n CALL api.enqueue_job(\n in_job_type := l_job.job_type,\n in_payload := l_job.payload,\n in_queue_name := l_job.queue_name,\n io_job_id := l_job_id\n );\n\n -- Update next run time (using pg_cron's cron parser or custom)\n UPDATE data.scheduled_jobs\n SET last_run_at = now(),\n next_run_at = private.next_cron_occurrence(l_job.cron_expression)\n WHERE id = l_job.id;\n END LOOP;\nEND;\n$;\n\n-- Schedule the scheduler (meta!)\nSELECT cron.schedule('run-scheduled-jobs', '* * * * *',\n $CALL private.run_scheduled_jobs()$);\n```\n\n### Delayed Jobs\n\n```sql\n-- Enqueue job for future execution\nCALL api.enqueue_job(\n in_job_type := 'send_reminder',\n in_payload := '{\"user_id\": \"...\"}'::jsonb,\n in_scheduled_at := now() + interval '1 day'\n);\n\n-- Enqueue job for specific time\nCALL api.enqueue_job(\n in_job_type := 'generate_report',\n in_payload := '{\"report_id\": \"...\"}'::jsonb,\n in_scheduled_at := '2024-04-01 09:00:00 UTC'\n);\n```\n\n## When to Use Dedicated Queues\n\n### Signs You Need a Dedicated Queue\n\n```markdown\n1. **High throughput**: > 1000 jobs/second\n2. **Complex routing**: Multiple consumers, topic-based routing\n3. **Priority queues**: Many priority levels needed\n4. **Message patterns**: Pub/sub, fan-out required\n5. **External producers**: Non-database systems need to enqueue\n6. **Acknowledgment patterns**: Need at-most-once or exactly-once guarantees\n```\n\n### Hybrid Approach\n\n```sql\n-- Use PostgreSQL for transactional jobs\n-- (Must complete with database transaction)\nBEGIN;\nINSERT INTO data.orders (customer_id, total) VALUES ($1, $2);\nCALL api.enqueue_job('send_confirmation', '{\"order_id\": \"...\"}'::jsonb);\nCOMMIT;\n\n-- Use Redis/RabbitMQ for high-volume, non-transactional jobs\n-- (Push to external queue from application)\n```\n\n### Queue Monitoring\n\n```sql\n-- Queue statistics view\nCREATE VIEW api.v_queue_stats AS\nSELECT\n queue_name,\n status,\n COUNT(*) AS job_count,\n MIN(scheduled_at) AS oldest_job,\n AVG(EXTRACT(EPOCH FROM (now() - scheduled_at))) AS avg_wait_seconds\nFROM data.job_queue\nWHERE status IN ('pending', 'processing')\nGROUP BY queue_name, status\n\nUNION ALL\n\nSELECT\n queue_name,\n 'completed_last_hour' AS status,\n COUNT(*) AS job_count,\n NULL AS oldest_job,\n AVG(EXTRACT(EPOCH FROM (completed_at - created_at))) AS avg_processing_seconds\nFROM data.job_queue\nWHERE status = 'completed'\n AND completed_at > now() - interval '1 hour'\nGROUP BY queue_name;\n\n-- Alert on queue backup\nCREATE FUNCTION app_monitoring.check_queue_health()\nRETURNS TABLE (queue_name text, status text, message text)\nLANGUAGE sql\nSTABLE\nAS $\n SELECT\n queue_name,\n CASE\n WHEN COUNT(*) > 1000 THEN 'CRITICAL'\n WHEN COUNT(*) > 100 THEN 'WARNING'\n ELSE 'OK'\n END,\n format('%s pending jobs, oldest: %s',\n COUNT(*),\n MIN(scheduled_at))\n FROM data.job_queue\n WHERE status = 'pending'\n GROUP BY queue_name;\n$;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":16671,"content_sha256":"575564390bbf44a7a4fb18cd1dc8c9b08856f7760da9eebfbdfd02ef94696a6c"},{"filename":"references/quick-reference.md","content":"# PostgreSQL Best Practices - Quick Reference Card\n\n> Print this page or keep it open while coding. For details, see the full documentation.\n\n---\n\n## Schema Architecture\n\n```\nApplication → api schema → data schema\n ↓\n private schema (triggers, helpers)\n```\n\n| Schema | Contains | Access |\n|--------|----------|--------|\n| `data` | Tables, indexes, constraints | None (internal) |\n| `private` | Triggers, helpers, password hashing | None (internal) |\n| `api` | Functions, procedures, views | Applications |\n| `app_audit` | Audit log tables | Admins only |\n| `app_migration` | Migration tracking | Admins only |\n\n**Data Warehouse Schemas** (if using Medallion Architecture):\n| `bronze` | Raw data landing | ETL role |\n| `silver` | Cleansed data | ETL role |\n| `gold` | Business-ready data | Analysts |\n| `dwh_lineage` | Data lineage tracking | ETL role |\n\n---\n\n## Trivadis Naming Conventions\n\n### Variables & Parameters\n\n| Prefix | Type | Example |\n|--------|------|---------|\n| `l_` | Local variable | `l_count`, `l_customer_id` |\n| `g_` | Session/global | `g_current_user_id` |\n| `co_` | Constant | `co_max_retries` |\n| `in_` | IN parameter | `in_customer_id` |\n| `out_` | OUT parameter (functions only) | `out_total` |\n| `io_` | INOUT parameter (procedures) | `io_id` |\n| `c_` | Cursor | `c_orders` |\n| `r_` | Record | `r_customer` |\n| `t_` | Array | `t_ids` |\n| `e_` | Exception | `e_not_found` |\n\n> **Note**: PostgreSQL procedures only support INOUT, not OUT. Use `io_` for procedure outputs.\n\n### Database Objects\n\n| Object | Pattern | Example |\n|--------|---------|---------|\n| Table | plural, snake_case | `customers`, `order_items` |\n| Column | singular, snake_case | `customer_id`, `created_at` |\n| PK | `{table}_pk` | `customers_pk` |\n| FK | `{table}_{ref}_fk` | `orders_customers_fk` |\n| Unique constraint | `{table}_{cols}_uk` | `customers_email_uk` |\n| Unique index | `{table}_{cols}_key` | `customers_email_key` |\n| Index | `{table}_{cols}_idx` | `orders_customer_id_idx` |\n| Check | `{table}_{col}_ck` | `orders_status_ck` |\n| Function | `{action}_{entity}` | `get_customer` |\n| Procedure | `{action}_{entity}` | `insert_order` |\n| Trigger | `{table}_{timing}{event}_trg` | `orders_bu_trg` |\n\n---\n\n## Data Types - Use / Avoid\n\n| ✅ Use | ❌ Avoid |\n|--------|----------|\n| `text` | `char(n)`, `varchar(n)` |\n| `numeric(p,s)` | `money`, `float`, `real` |\n| `timestamptz` | `timestamp` |\n| `boolean` | `integer` flags |\n| `uuid DEFAULT uuidv7()` | `serial`, `uuid_generate_v4()` |\n| `GENERATED ALWAYS AS IDENTITY` | `serial`, `bigserial` |\n| `jsonb` | `json`, EAV tables |\n| `integer` / `bigint` | `smallint` (unless space-critical) |\n\n---\n\n## Essential Patterns\n\n### Create Table\n```sql\nCREATE TABLE data.customers (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n email text NOT NULL,\n name text NOT NULL,\n is_active boolean NOT NULL DEFAULT true,\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE UNIQUE INDEX customers_email_key ON data.customers(lower(email));\nCREATE INDEX customers_is_active_idx ON data.customers(is_active) WHERE is_active;\n\nCREATE TRIGGER customers_bu_trg\n BEFORE UPDATE ON data.customers\n FOR EACH ROW EXECUTE FUNCTION private.set_updated_at();\n```\n\n### API Function (Read)\n```sql\nCREATE FUNCTION api.get_customer(in_id uuid)\nRETURNS TABLE (id uuid, email text, name text)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name FROM data.customers WHERE id = in_id;\n$;\n```\n\n### API Procedure (Write)\n```sql\nCREATE PROCEDURE api.insert_customer(\n in_email text,\n in_name text,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_email text;\nBEGIN\n l_email := lower(trim(in_email));\n \n INSERT INTO data.customers (email, name)\n VALUES (l_email, trim(in_name))\n RETURNING id INTO io_id;\nEND;\n$;\n```\n\n### Private Trigger Function\n```sql\nCREATE FUNCTION private.set_updated_at()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nBEGIN\n NEW.updated_at := now();\n RETURN NEW;\nEND;\n$;\n```\n\n### Error Handling\n```sql\n-- Raise custom error\nRAISE EXCEPTION 'Customer not found: %', in_id\n USING ERRCODE = 'P0001';\n\n-- Handle errors\nBEGIN\n -- risky operation\nEXCEPTION\n WHEN unique_violation THEN\n RAISE EXCEPTION 'Email already exists' USING ERRCODE = 'P0002';\n WHEN OTHERS THEN\n RAISE; -- Re-raise unexpected errors\nEND;\n```\n\n---\n\n## Index Quick Reference\n\n| Query Pattern | Index Type |\n|---------------|------------|\n| `=`, `\u003c`, `>`, `BETWEEN`, `ORDER BY` | B-tree (default) |\n| `=` only (large table) | Hash |\n| `LIKE 'prefix%'` | B-tree |\n| `LIKE '%text%'` | GIN + pg_trgm |\n| `@>`, `?`, `?&` (JSONB) | GIN |\n| `@@` (full-text) | GIN |\n| Geometry, ranges | GiST |\n| Very large, ordered data | BRIN |\n\n```sql\n-- Always index foreign keys!\nCREATE INDEX orders_customer_id_idx ON data.orders(customer_id);\n\n-- Partial index for common queries\nCREATE INDEX orders_pending_idx ON data.orders(created_at)\n WHERE status = 'pending';\n\n-- Covering index (includes extra columns)\nCREATE INDEX orders_status_idx ON data.orders(status)\n INCLUDE (total, created_at);\n```\n\n---\n\n## Migrations Quick Reference\n\n```sql\n-- 1. Acquire lock\nSELECT app_migration.acquire_lock();\n\n-- 2. Run versioned migration (runs once)\nCALL app_migration.run_versioned(\n in_version := '001',\n in_description := 'Create customers table',\n in_sql := $mig$ CREATE TABLE data.customers (...); $mig$,\n in_rollback_sql := 'DROP TABLE IF EXISTS data.customers;'\n);\n\n-- 3. Run repeatable migration (re-runs if changed)\nCALL app_migration.run_repeatable(\n in_filename := 'R__api_functions.sql',\n in_description := 'API functions',\n in_sql := $mig$ CREATE OR REPLACE FUNCTION api.get_customer... $mig$\n);\n\n-- 4. Release lock\nSELECT app_migration.release_lock();\n```\n\n---\n\n## Connection Pool Sizing\n\n```\npool_size = (CPU cores × 2) + spindle_count -- HDD\npool_size = CPU cores × 2 -- SSD / cloud\n```\n\n| Cores | Pool Size | Notes |\n|-------|-----------|-------|\n| 4 | 8 | Small cloud instance |\n| 8 | 16 | Medium |\n| 16 | 32 | Large (SSD) |\n\nKeep `max_connections` low (2-4× pool size) when using PgBouncer. Each idle connection costs ~5-10 MB RAM.\n\n---\n\n## Grants / Permissions\n\n```sql\n-- Create role\nCREATE ROLE app_service LOGIN PASSWORD 'secure';\n\n-- Grant API access only\nGRANT USAGE ON SCHEMA api TO app_service;\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA api TO app_service;\nGRANT EXECUTE ON ALL PROCEDURES IN SCHEMA api TO app_service;\n\n-- Set search path\nALTER ROLE app_service SET search_path = api, pg_temp;\n\n-- NEVER grant on data or private schemas!\n```\n\n---\n\n## Critical Anti-Patterns\n\n| ❌ Don't | ✅ Do |\n|----------|-------|\n| `RETURNS SETOF table` | `RETURNS TABLE (col1 type, ...)` |\n| `SECURITY DEFINER` without `SET search_path` | Always include both |\n| `SELECT *` | Explicit column list |\n| `NOT IN (subquery)` | `NOT EXISTS (...)` |\n| `BETWEEN` with timestamps | `>= AND \u003c` |\n| Missing FK indexes | Always index FKs |\n| `timestamp` | `timestamptz` |\n| `varchar(255)` | `text` |\n| N+1 loops (1 query + N per row) | Batch fetch with `ANY(uuid[])`, JOINs, or `LATERAL` |\n\n---\n\n## PostgreSQL 18+ Features\n\n```sql\n-- UUIDv7 (timestamp-ordered)\nid uuid PRIMARY KEY DEFAULT uuidv7()\n\n-- Extract timestamp from UUIDv7\nSELECT uuid_extract_timestamp(id) FROM data.orders;\n\n-- Virtual generated column\nfull_name text GENERATED ALWAYS AS (first_name || ' ' || last_name) VIRTUAL\n\n-- OLD/NEW in RETURNING\nUPDATE data.orders SET status = 'shipped'\nRETURNING OLD.status AS old_status, NEW.status AS new_status;\n```\n\n---\n\n## Common SQL Patterns\n\n```sql\n-- Upsert\nINSERT INTO data.customers (email, name) VALUES ($1, $2)\nON CONFLICT (email) DO UPDATE SET name = EXCLUDED.name;\n\n-- Batch insert\nINSERT INTO data.orders (customer_id, total)\nSELECT unnest($1::uuid[]), unnest($2::numeric[]);\n\n-- Pagination\nSELECT * FROM data.orders\nORDER BY created_at DESC\nLIMIT 20 OFFSET 40; -- Page 3, 20 per page\n\n-- Conditional update\nUPDATE data.customers\nSET name = COALESCE(in_name, name),\n email = COALESCE(in_email, email)\nWHERE id = in_id;\n```\n\n---\n\n*For complete documentation, see SKILL.md and reference files.*\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":8376,"content_sha256":"0747dc0255e8c5b76d2d413dc33afaabe2dd243afdecd3e365e05e796dbb2438"},{"filename":"references/replication-ha.md","content":"# Replication & High Availability\n\nThis document covers PostgreSQL replication strategies including streaming replication, logical replication, and high availability configurations.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Streaming Replication](#streaming-replication)\n3. [Logical Replication](#logical-replication)\n4. [Synchronous Replication](#synchronous-replication)\n5. [Read Replica Patterns](#read-replica-patterns)\n6. [Failover Strategies](#failover-strategies)\n7. [Connection Routing](#connection-routing)\n8. [Monitoring Replication](#monitoring-replication)\n9. [Common Configurations](#common-configurations)\n\n## Overview\n\n### Replication Types Comparison\n\n| Feature | Streaming Replication | Logical Replication |\n|---------|----------------------|---------------------|\n| Granularity | Full cluster | Per-table |\n| Cross-version | No | Yes |\n| Selective tables | No | Yes |\n| Write to replica | No | Yes (different tables) |\n| Performance | Better | Good |\n| Setup complexity | Lower | Higher |\n| Use case | HA, read scaling | Data distribution, upgrades |\n\n### High Availability Architecture\n\n```mermaid\nflowchart TB\n subgraph CLIENTS[\"Clients\"]\n APP1[\"App Server 1\"]\n APP2[\"App Server 2\"]\n end\n\n subgraph LB[\"Load Balancer / Proxy\"]\n PGBOUNCER[\"PgBouncer / HAProxy\"]\n end\n\n subgraph CLUSTER[\"PostgreSQL Cluster\"]\n PRIMARY[(\"Primary\u003cbr/>Read/Write\")]\n SYNC[(\"Sync Replica\u003cbr/>Read Only\")]\n ASYNC1[(\"Async Replica 1\u003cbr/>Read Only\")]\n ASYNC2[(\"Async Replica 2\u003cbr/>Read Only\")]\n end\n\n APP1 --> PGBOUNCER\n APP2 --> PGBOUNCER\n\n PGBOUNCER -->|writes| PRIMARY\n PGBOUNCER -->|reads| SYNC\n PGBOUNCER -->|reads| ASYNC1\n PGBOUNCER -->|reads| ASYNC2\n\n PRIMARY -.->|sync| SYNC\n PRIMARY -.->|async| ASYNC1\n PRIMARY -.->|async| ASYNC2\n\n style PRIMARY fill:#c8e6c9\n style SYNC fill:#bbdefb\n style ASYNC1 fill:#fff3e0\n style ASYNC2 fill:#fff3e0\n```\n\n## Streaming Replication\n\n### Primary Server Configuration\n\n```sql\n-- postgresql.conf on PRIMARY\n\n# Replication settings\nwal_level = replica # Required for replication\nmax_wal_senders = 10 # Max replication connections\nwal_keep_size = 1GB # WAL to keep for slow replicas\nmax_replication_slots = 10 # Replication slots\n\n# Optional: Hot standby feedback\nhot_standby_feedback = on # Prevent vacuum conflicts\n\n# Archive settings (for PITR)\narchive_mode = on\narchive_command = 'cp %p /var/backups/wal/%f'\n```\n\n```sql\n-- pg_hba.conf on PRIMARY\n# Allow replication connections\nhost replication replicator 10.0.0.0/8 scram-sha-256\nhost replication replicator replica1.example.com scram-sha-256\n```\n\n### Create Replication User\n\n```sql\n-- On PRIMARY\nCREATE USER replicator WITH REPLICATION ENCRYPTED PASSWORD 'secure_password';\n\n-- Grant necessary permissions\nGRANT pg_read_all_data TO replicator;\n```\n\n### Create Replication Slot\n\n```sql\n-- On PRIMARY: Create slot to prevent WAL removal\nSELECT pg_create_physical_replication_slot('replica1_slot');\n\n-- View slots\nSELECT slot_name, slot_type, active, restart_lsn\nFROM pg_replication_slots;\n```\n\n### Setup Replica Server\n\n```bash\n# On REPLICA: Stop PostgreSQL\nsudo systemctl stop postgresql\n\n# Remove existing data\nsudo rm -rf /var/lib/postgresql/data/*\n\n# Base backup from primary\npg_basebackup \\\n -h primary.example.com \\\n -U replicator \\\n -D /var/lib/postgresql/data \\\n -P \\\n --wal-method=stream \\\n --slot=replica1_slot \\\n --write-recovery-conf\n\n# Fix ownership\nsudo chown -R postgres:postgres /var/lib/postgresql/data\n\n# Start PostgreSQL\nsudo systemctl start postgresql\n```\n\n### Replica Configuration\n\n```sql\n-- postgresql.conf on REPLICA\n\n# Standby settings\nhot_standby = on # Allow read queries\nprimary_conninfo = 'host=primary.example.com port=5432 user=replicator password=secure_password'\nprimary_slot_name = 'replica1_slot'\n\n# Performance tuning\nmax_standby_streaming_delay = 30s # Max delay before canceling queries\nmax_standby_archive_delay = 300s # Max delay for archive recovery\nwal_receiver_timeout = 60s # Timeout for WAL receiver\n```\n\n### Verify Replication\n\n```sql\n-- On PRIMARY: Check replication status\nSELECT\n client_addr,\n state,\n sent_lsn,\n write_lsn,\n flush_lsn,\n replay_lsn,\n sync_state\nFROM pg_stat_replication;\n\n-- On REPLICA: Check standby status\nSELECT\n pg_is_in_recovery() AS is_replica,\n pg_last_wal_receive_lsn() AS receive_lsn,\n pg_last_wal_replay_lsn() AS replay_lsn,\n pg_last_xact_replay_timestamp() AS last_replay_time,\n now() - pg_last_xact_replay_timestamp() AS replication_lag;\n```\n\n## Logical Replication\n\n### Publisher Configuration\n\n```sql\n-- postgresql.conf on PUBLISHER\nwal_level = logical\nmax_replication_slots = 10\nmax_wal_senders = 10\n```\n\n### Create Publication\n\n```sql\n-- On PUBLISHER: Create publication for specific tables\nCREATE PUBLICATION my_publication\n FOR TABLE data.orders, data.customers;\n\n-- Or for all tables in schema\nCREATE PUBLICATION my_publication\n FOR TABLES IN SCHEMA data;\n\n-- Or for all tables\nCREATE PUBLICATION my_publication\n FOR ALL TABLES;\n\n-- With options\nCREATE PUBLICATION my_publication\n FOR TABLE data.orders, data.customers\n WITH (publish = 'insert, update, delete');\n\n-- View publications\nSELECT * FROM pg_publication;\nSELECT * FROM pg_publication_tables;\n```\n\n### Subscriber Configuration\n\n```sql\n-- On SUBSCRIBER: Create the tables first (same structure)\n-- Tables must exist before subscription\n\n-- Create subscription\nCREATE SUBSCRIPTION my_subscription\n CONNECTION 'host=publisher.example.com port=5432 dbname=mydb user=replicator password=xxx'\n PUBLICATION my_publication;\n\n-- With options\nCREATE SUBSCRIPTION my_subscription\n CONNECTION 'host=publisher.example.com port=5432 dbname=mydb user=replicator password=xxx'\n PUBLICATION my_publication\n WITH (\n copy_data = true, -- Initial data sync\n create_slot = true, -- Create replication slot\n enabled = true, -- Start immediately\n synchronous_commit = 'off' -- Performance tuning\n );\n\n-- View subscriptions\nSELECT * FROM pg_subscription;\nSELECT * FROM pg_stat_subscription;\n```\n\n### Manage Logical Replication\n\n```sql\n-- Add table to publication\nALTER PUBLICATION my_publication ADD TABLE data.new_table;\n\n-- Remove table from publication\nALTER PUBLICATION my_publication DROP TABLE data.old_table;\n\n-- Refresh subscription (after adding tables)\nALTER SUBSCRIPTION my_subscription REFRESH PUBLICATION;\n\n-- Disable/enable subscription\nALTER SUBSCRIPTION my_subscription DISABLE;\nALTER SUBSCRIPTION my_subscription ENABLE;\n\n-- Drop subscription\nDROP SUBSCRIPTION my_subscription;\n```\n\n### Logical Replication for Zero-Downtime Upgrades\n\n```sql\n-- Step 1: Setup logical replication from old to new version\n-- On NEW (subscriber)\nCREATE SUBSCRIPTION upgrade_sub\n CONNECTION 'host=old-server dbname=mydb user=replicator password=xxx'\n PUBLICATION all_tables;\n\n-- Step 2: Wait for initial sync\nSELECT * FROM pg_stat_subscription;\n-- Wait until srsubstate = 'r' (ready) for all tables\n\n-- Step 3: Verify data sync\n-- Compare row counts between old and new\n\n-- Step 4: Switch applications to new server\n\n-- Step 5: Drop subscription and publication\n-- On NEW\nDROP SUBSCRIPTION upgrade_sub;\n-- On OLD\nDROP PUBLICATION all_tables;\n```\n\n## Synchronous Replication\n\n### Configure Synchronous Standby\n\n```sql\n-- postgresql.conf on PRIMARY\n\n# Synchronous replication\nsynchronous_standby_names = 'FIRST 1 (replica1, replica2)'\nsynchronous_commit = on\n\n# Options:\n# 'replica1' - Specific standby\n# 'FIRST 1 (r1, r2, r3)' - First to respond\n# 'ANY 2 (r1, r2, r3)' - Any 2 must confirm\n# '*' - Any standby\n```\n\n### Synchronous Commit Levels\n\n```sql\n-- Per-transaction override\nSET synchronous_commit = 'off'; -- No durability (fastest)\nSET synchronous_commit = 'local'; -- Local WAL flush only\nSET synchronous_commit = 'remote_write';-- Remote received\nSET synchronous_commit = 'on'; -- Remote WAL flush (default)\nSET synchronous_commit = 'remote_apply';-- Remote WAL applied\n\n-- For specific transactions\nBEGIN;\nSET LOCAL synchronous_commit = 'remote_apply';\nINSERT INTO critical_data ...;\nCOMMIT;\n```\n\n### Application Name for Sync Identification\n\n```sql\n-- On REPLICA: Set application name\n-- primary_conninfo in postgresql.conf or recovery.conf\nprimary_conninfo = 'host=primary port=5432 user=replicator application_name=replica1'\n\n-- Or via connection string\n-- postgresql://replicator@primary:5432/mydb?application_name=replica1\n```\n\n## Read Replica Patterns\n\n### Query Routing in Application\n\n```sql\n-- API function that reads from replica\nCREATE FUNCTION api.select_orders_readonly(in_customer_id uuid)\nRETURNS TABLE (\n id uuid,\n total numeric,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n -- This function can run on replica\n SELECT id, total, created_at\n FROM data.orders\n WHERE customer_id = in_customer_id\n ORDER BY created_at DESC;\n$;\n```\n\n### Handle Replication Lag\n\n```sql\n-- Check acceptable lag before querying replica\nCREATE FUNCTION api.select_with_lag_check(\n in_max_lag_seconds integer DEFAULT 5\n)\nRETURNS TABLE (...)\nLANGUAGE plpgsql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_lag interval;\nBEGIN\n -- Check replication lag\n SELECT now() - pg_last_xact_replay_timestamp()\n INTO l_lag;\n\n IF l_lag > make_interval(secs := in_max_lag_seconds) THEN\n RAISE EXCEPTION 'Replication lag too high: %', l_lag\n USING ERRCODE = 'P0001';\n END IF;\n\n RETURN QUERY SELECT ...;\nEND;\n$;\n```\n\n### Read Your Own Writes Pattern\n\n```sql\n-- Track last write LSN for session\nCREATE TABLE private.session_write_lsn (\n session_id text PRIMARY KEY,\n last_lsn pg_lsn NOT NULL,\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- After write on primary\nCREATE FUNCTION private.record_write_lsn()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nBEGIN\n INSERT INTO private.session_write_lsn (session_id, last_lsn)\n VALUES (current_setting('app.session_id'), pg_current_wal_lsn())\n ON CONFLICT (session_id) DO UPDATE\n SET last_lsn = EXCLUDED.last_lsn, updated_at = now();\n RETURN NULL;\nEND;\n$;\n\n-- On replica: wait for LSN before reading\nCREATE FUNCTION api.wait_for_lsn(in_session_id text, in_timeout_ms integer DEFAULT 5000)\nRETURNS boolean\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_target_lsn pg_lsn;\nBEGIN\n -- Get target LSN from session tracking\n SELECT last_lsn INTO l_target_lsn\n FROM private.session_write_lsn\n WHERE session_id = in_session_id;\n\n IF l_target_lsn IS NULL THEN\n RETURN true; -- No write to wait for\n END IF;\n\n -- Wait for replay to catch up\n RETURN pg_wal_replay_wait(l_target_lsn, in_timeout_ms);\nEND;\n$;\n```\n\n## Failover Strategies\n\n### Manual Failover\n\n```sql\n-- On REPLICA: Promote to primary\nSELECT pg_promote();\n\n-- Or via command line\npg_ctl promote -D /var/lib/postgresql/data\n\n-- Or create trigger file (legacy)\n-- touch /var/lib/postgresql/data/standby.signal\n```\n\n### Automatic Failover with Patroni\n\n```yaml\n# patroni.yml\nscope: postgres-cluster\nname: node1\n\nrestapi:\n listen: 0.0.0.0:8008\n connect_address: node1.example.com:8008\n\netcd:\n hosts: etcd1:2379,etcd2:2379,etcd3:2379\n\nbootstrap:\n dcs:\n ttl: 30\n loop_wait: 10\n retry_timeout: 10\n maximum_lag_on_failover: 1048576\n postgresql:\n use_pg_rewind: true\n parameters:\n wal_level: replica\n hot_standby: on\n max_wal_senders: 10\n max_replication_slots: 10\n\n initdb:\n - encoding: UTF8\n - data-checksums\n\npostgresql:\n listen: 0.0.0.0:5432\n connect_address: node1.example.com:5432\n data_dir: /var/lib/postgresql/data\n authentication:\n superuser:\n username: postgres\n password: xxx\n replication:\n username: replicator\n password: xxx\n```\n\n### Repmgr Failover\n\n```bash\n# repmgr.conf\nnode_id=1\nnode_name='node1'\nconninfo='host=node1.example.com user=repmgr dbname=repmgr'\ndata_directory='/var/lib/postgresql/data'\nfailover='automatic'\npromote_command='/usr/bin/repmgr standby promote -f /etc/repmgr.conf'\nfollow_command='/usr/bin/repmgr standby follow -f /etc/repmgr.conf'\n```\n\n### Failover Checklist\n\n```markdown\n## Failover Procedure\n\n### Pre-Failover\n- [ ] Verify primary is truly unavailable\n- [ ] Check replica sync status\n- [ ] Notify stakeholders\n\n### Failover Steps\n1. [ ] Promote replica to primary\n ```sql\n SELECT pg_promote();\n ```\n\n2. [ ] Update connection strings/DNS\n3. [ ] Verify new primary accepts writes\n4. [ ] Update monitoring\n\n### Post-Failover\n- [ ] Investigate old primary failure\n- [ ] Rebuild old primary as replica (if recoverable)\n- [ ] Update documentation\n- [ ] Post-mortem report\n```\n\n## Connection Routing\n\n### HAProxy Configuration\n\n```\n# haproxy.cfg\nglobal\n maxconn 1000\n\ndefaults\n mode tcp\n timeout connect 10s\n timeout client 30m\n timeout server 30m\n\nlisten postgres-primary\n bind *:5432\n option httpchk GET /primary\n http-check expect status 200\n default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions\n server node1 node1:5432 check port 8008\n server node2 node2:5432 check port 8008\n server node3 node3:5432 check port 8008\n\nlisten postgres-replica\n bind *:5433\n balance roundrobin\n option httpchk GET /replica\n http-check expect status 200\n default-server inter 3s fall 3 rise 2\n server node1 node1:5432 check port 8008\n server node2 node2:5432 check port 8008\n server node3 node3:5432 check port 8008\n```\n\n### PgBouncer with Multiple Databases\n\n```ini\n# pgbouncer.ini\n[databases]\n# Write database (primary)\nmydb = host=primary.example.com port=5432 dbname=mydb\n\n# Read database (replica)\nmydb_ro = host=replica.example.com port=5432 dbname=mydb\n\n[pgbouncer]\nlisten_port = 6432\nlisten_addr = *\nauth_type = scram-sha-256\nauth_file = /etc/pgbouncer/userlist.txt\npool_mode = transaction\nmax_client_conn = 1000\ndefault_pool_size = 20\n```\n\n### Application Connection String\n\n```python\n# Python example with read/write splitting\nimport psycopg\n\n# Write connection\nwrite_conn = psycopg.connect(\n \"host=pgbouncer port=6432 dbname=mydb user=app\"\n)\n\n# Read connection (to replica)\nread_conn = psycopg.connect(\n \"host=pgbouncer port=6432 dbname=mydb_ro user=app\"\n)\n\n# Or using target_session_attrs\nauto_conn = psycopg.connect(\n \"host=node1,node2,node3 port=5432 dbname=mydb target_session_attrs=read-write\"\n)\n```\n\n## Monitoring Replication\n\n### Replication Lag Monitoring\n\n```sql\n-- On PRIMARY: Check all replicas\nSELECT\n application_name,\n client_addr,\n state,\n sync_state,\n pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes,\n pg_wal_lsn_diff(pg_current_wal_lsn(), flush_lsn) AS flush_lag_bytes,\n pg_wal_lsn_diff(pg_current_wal_lsn(), write_lsn) AS write_lag_bytes\nFROM pg_stat_replication;\n\n-- On REPLICA: Check lag\nSELECT\n CASE\n WHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() THEN 0\n ELSE EXTRACT(EPOCH FROM now() - pg_last_xact_replay_timestamp())\n END AS lag_seconds;\n```\n\n### Create Monitoring View\n\n```sql\nCREATE VIEW api.v_replication_status AS\nSELECT\n 'primary' AS role,\n NULL AS receive_lsn,\n pg_current_wal_lsn() AS current_lsn,\n NULL AS replay_lsn,\n NULL AS lag_seconds\nWHERE NOT pg_is_in_recovery()\n\nUNION ALL\n\nSELECT\n 'replica' AS role,\n pg_last_wal_receive_lsn() AS receive_lsn,\n NULL AS current_lsn,\n pg_last_wal_replay_lsn() AS replay_lsn,\n EXTRACT(EPOCH FROM now() - pg_last_xact_replay_timestamp()) AS lag_seconds\nWHERE pg_is_in_recovery();\n```\n\n### Alerting on Replication Issues\n\n```sql\n-- Function to check replication health\nCREATE FUNCTION app_monitoring.check_replication_health()\nRETURNS TABLE (\n check_name text,\n status text,\n message text\n)\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Check if primary\n IF NOT pg_is_in_recovery() THEN\n -- Check connected replicas\n RETURN QUERY\n SELECT\n 'replica_connected'::text,\n CASE WHEN count(*) > 0 THEN 'OK' ELSE 'WARNING' END,\n format('%s replicas connected', count(*))\n FROM pg_stat_replication;\n\n -- Check for lagging replicas\n RETURN QUERY\n SELECT\n 'replica_lag'::text,\n CASE\n WHEN max(pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)) > 100000000\n THEN 'CRITICAL'\n WHEN max(pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)) > 10000000\n THEN 'WARNING'\n ELSE 'OK'\n END,\n format('Max lag: %s bytes', max(pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)))\n FROM pg_stat_replication;\n ELSE\n -- Check replica lag\n RETURN QUERY\n SELECT\n 'replication_lag'::text,\n CASE\n WHEN extract(epoch from now() - pg_last_xact_replay_timestamp()) > 60\n THEN 'CRITICAL'\n WHEN extract(epoch from now() - pg_last_xact_replay_timestamp()) > 10\n THEN 'WARNING'\n ELSE 'OK'\n END,\n format('Lag: %s seconds',\n round(extract(epoch from now() - pg_last_xact_replay_timestamp())::numeric, 2));\n END IF;\nEND;\n$;\n```\n\n## Common Configurations\n\n### Two-Node HA (Async)\n\n```\nPrimary \u003c--(async)--> Replica (hot standby)\n\n- Simple setup\n- Potential data loss on failover\n- Use for non-critical data\n```\n\n### Three-Node HA (Sync)\n\n```\nPrimary \u003c--(sync)--> Sync Replica\n ^\n |\n └--(async)--> Async Replica\n\n- No data loss with sync replica\n- Async replica for read scaling\n- Use for critical data\n```\n\n### Multi-Region\n\n```\nRegion A:\n Primary \u003c--(sync)--> Local Replica\n\nRegion B:\n Primary(A) \u003c--(async)--> DR Replica\n\n- Local sync for zero data loss\n- Async to DR for disaster recovery\n- Accept lag for cross-region\n```\n\n### Read Scaling\n\n```\nPrimary\n |\n ├--(async)--> Replica 1 (reads)\n ├--(async)--> Replica 2 (reads)\n └--(async)--> Replica 3 (reporting)\n\n- Multiple replicas for read load\n- Dedicated replica for heavy queries\n- Load balancer distributes reads\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":18401,"content_sha256":"fdf5b1698d0ef1df73977de6d6791c1115edcfad06c9c8604eabc59688fe7864"},{"filename":"references/row-level-security.md","content":"# Row-Level Security (RLS) Patterns\n\nRow-Level Security provides fine-grained access control at the row level, complementing our schema separation pattern. This document covers multi-tenant isolation, user-specific access, and integration with SECURITY DEFINER functions.\n\n## Table of Contents\n\n1. [RLS Fundamentals](#rls-fundamentals)\n2. [Multi-Tenant Patterns](#multi-tenant-patterns)\n3. [User-Specific Access](#user-specific-access)\n4. [RLS with SECURITY DEFINER](#rls-with-security-definer)\n5. [Policy Patterns](#policy-patterns)\n6. [Performance Considerations](#performance-considerations)\n7. [Testing RLS](#testing-rls)\n8. [Common Pitfalls](#common-pitfalls)\n\n## RLS Fundamentals\n\n### Enabling RLS\n\n```sql\n-- Enable RLS on a table\nALTER TABLE data.orders ENABLE ROW LEVEL SECURITY;\n\n-- Force RLS for table owner too (important for testing)\nALTER TABLE data.orders FORCE ROW LEVEL SECURITY;\n\n-- Check RLS status\nSELECT \n tablename,\n rowsecurity,\n forcerowsecurity\nFROM pg_tables\nWHERE schemaname = 'data';\n```\n\n### Basic Policy Structure\n\n```sql\n-- Policy syntax\nCREATE POLICY policy_name\n ON schema.table\n AS {PERMISSIVE | RESTRICTIVE} -- Default: PERMISSIVE\n FOR {ALL | SELECT | INSERT | UPDATE | DELETE}\n TO {role_name | PUBLIC | CURRENT_USER}\n USING (condition_for_existing_rows) -- For SELECT, UPDATE, DELETE\n WITH CHECK (condition_for_new_rows); -- For INSERT, UPDATE\n```\n\n### Policy Types\n\n```mermaid\ngraph TD\n subgraph \"PERMISSIVE Policies (OR)\"\n P1[\"Policy 1: tenant_id = current_tenant()\"]\n P2[\"Policy 2: is_public = true\"]\n end\n \n subgraph \"RESTRICTIVE Policies (AND)\"\n R1[\"Policy: NOT is_deleted\"]\n end\n \n P1 -->|OR| RESULT[\"Row Accessible\"]\n P2 -->|OR| RESULT\n RESULT -->|AND| R1\n R1 --> FINAL[\"Final Access\"]\n```\n\n```sql\n-- Multiple PERMISSIVE policies are OR'd together\n-- RESTRICTIVE policies are AND'd with the result\n\n-- Example: User can see own data OR public data\nCREATE POLICY see_own_data ON data.documents\n AS PERMISSIVE\n FOR SELECT\n USING (owner_id = current_user_id());\n\nCREATE POLICY see_public_data ON data.documents\n AS PERMISSIVE\n FOR SELECT\n USING (is_public = true);\n\n-- Must also pass: not deleted (RESTRICTIVE)\nCREATE POLICY hide_deleted ON data.documents\n AS RESTRICTIVE\n FOR SELECT\n USING (NOT is_deleted);\n```\n\n## Multi-Tenant Patterns\n\n### Session-Based Tenant Context\n\n```sql\n-- Set tenant context at connection/transaction start\nCREATE OR REPLACE FUNCTION private.set_tenant(in_tenant_id uuid)\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n PERFORM set_config('app.current_tenant_id', in_tenant_id::text, false);\nEND;\n$;\n\n-- Get current tenant (used in policies)\nCREATE OR REPLACE FUNCTION private.current_tenant_id()\nRETURNS uuid\nLANGUAGE sql\nSTABLE\nAS $\n SELECT NULLIF(current_setting('app.current_tenant_id', true), '')::uuid;\n$;\n\n-- Tables with tenant_id\nCREATE TABLE data.customers (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n tenant_id uuid NOT NULL,\n email text NOT NULL,\n name text NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Enable RLS\nALTER TABLE data.customers ENABLE ROW LEVEL SECURITY;\nALTER TABLE data.customers FORCE ROW LEVEL SECURITY;\n\n-- Tenant isolation policy\nCREATE POLICY tenant_isolation ON data.customers\n FOR ALL\n USING (tenant_id = private.current_tenant_id())\n WITH CHECK (tenant_id = private.current_tenant_id());\n\n-- Index for policy performance\nCREATE INDEX customers_tenant_idx ON data.customers(tenant_id);\n```\n\n### API Functions with Tenant Context\n\n```sql\n-- Option 1: Tenant passed explicitly\nCREATE FUNCTION api.get_customer(in_tenant_id uuid, in_customer_id uuid)\nRETURNS TABLE (id uuid, email text, name text)\nLANGUAGE plpgsql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n -- Set tenant context for RLS\n PERFORM set_config('app.current_tenant_id', in_tenant_id::text, true);\n \n RETURN QUERY\n SELECT c.id, c.email, c.name\n FROM data.customers c\n WHERE c.id = in_customer_id;\n -- RLS automatically filters by tenant_id\nEND;\n$;\n\n-- Option 2: Tenant from session (set by connection middleware)\nCREATE FUNCTION api.get_customer(in_customer_id uuid)\nRETURNS TABLE (id uuid, email text, name text)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name\n FROM data.customers\n WHERE id = in_customer_id;\n -- RLS uses current_setting('app.current_tenant_id')\n$;\n```\n\n### Tenant Hierarchy (Parent/Child Tenants)\n\n```sql\n-- Tenant table with hierarchy\nCREATE TABLE data.tenants (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n parent_id uuid REFERENCES data.tenants(id),\n name text NOT NULL,\n is_active boolean NOT NULL DEFAULT true\n);\n\n-- Function to get accessible tenant IDs (self + children)\nCREATE OR REPLACE FUNCTION private.accessible_tenant_ids()\nRETURNS uuid[]\nLANGUAGE sql\nSTABLE\nAS $\n WITH RECURSIVE tenant_tree AS (\n SELECT id FROM data.tenants \n WHERE id = private.current_tenant_id()\n \n UNION ALL\n \n SELECT t.id FROM data.tenants t\n JOIN tenant_tree tt ON t.parent_id = tt.id\n )\n SELECT array_agg(id) FROM tenant_tree;\n$;\n\n-- Policy allowing access to child tenant data\nCREATE POLICY tenant_hierarchy ON data.orders\n FOR SELECT\n USING (tenant_id = ANY(private.accessible_tenant_ids()));\n```\n\n## User-Specific Access\n\n### Basic User Ownership\n\n```sql\nCREATE TABLE data.documents (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n owner_id uuid NOT NULL REFERENCES data.users(id),\n title text NOT NULL,\n content text,\n is_public boolean NOT NULL DEFAULT false,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\nALTER TABLE data.documents ENABLE ROW LEVEL SECURITY;\n\n-- Get current user from session\nCREATE OR REPLACE FUNCTION private.current_user_id()\nRETURNS uuid\nLANGUAGE sql\nSTABLE\nAS $\n SELECT NULLIF(current_setting('app.current_user_id', true), '')::uuid;\n$;\n\n-- Owner can do everything\nCREATE POLICY owner_all ON data.documents\n FOR ALL\n USING (owner_id = private.current_user_id())\n WITH CHECK (owner_id = private.current_user_id());\n\n-- Anyone can read public documents\nCREATE POLICY public_read ON data.documents\n FOR SELECT\n USING (is_public = true);\n```\n\n### Role-Based Access\n\n```sql\n-- Users table with role\nCREATE TABLE data.users (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n tenant_id uuid NOT NULL,\n email text NOT NULL,\n role text NOT NULL DEFAULT 'user', -- 'admin', 'manager', 'user'\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Get current user's role\nCREATE OR REPLACE FUNCTION private.current_user_role()\nRETURNS text\nLANGUAGE sql\nSTABLE\nAS $\n SELECT role FROM data.users WHERE id = private.current_user_id();\n$;\n\n-- Admins can see all within tenant\nCREATE POLICY admin_all ON data.orders\n FOR ALL\n USING (\n tenant_id = private.current_tenant_id()\n AND private.current_user_role() = 'admin'\n );\n\n-- Managers can see their team's orders\nCREATE POLICY manager_team ON data.orders\n FOR SELECT\n USING (\n tenant_id = private.current_tenant_id()\n AND private.current_user_role() = 'manager'\n AND created_by IN (\n SELECT id FROM data.users \n WHERE manager_id = private.current_user_id()\n )\n );\n\n-- Users can only see own orders\nCREATE POLICY user_own ON data.orders\n FOR ALL\n USING (\n tenant_id = private.current_tenant_id()\n AND created_by = private.current_user_id()\n )\n WITH CHECK (\n tenant_id = private.current_tenant_id()\n AND created_by = private.current_user_id()\n );\n```\n\n### Shared Access (Many-to-Many)\n\n```sql\n-- Document sharing table\nCREATE TABLE data.document_shares (\n document_id uuid NOT NULL REFERENCES data.documents(id) ON DELETE CASCADE,\n user_id uuid NOT NULL REFERENCES data.users(id) ON DELETE CASCADE,\n permission text NOT NULL DEFAULT 'read', -- 'read', 'write', 'admin'\n created_at timestamptz NOT NULL DEFAULT now(),\n PRIMARY KEY (document_id, user_id)\n);\n\n-- Policy: owner OR shared with me\nCREATE POLICY can_read ON data.documents\n FOR SELECT\n USING (\n owner_id = private.current_user_id()\n OR is_public = true\n OR EXISTS (\n SELECT 1 FROM data.document_shares\n WHERE document_id = documents.id\n AND user_id = private.current_user_id()\n )\n );\n\n-- Policy: owner OR shared with write permission\nCREATE POLICY can_write ON data.documents\n FOR UPDATE\n USING (\n owner_id = private.current_user_id()\n OR EXISTS (\n SELECT 1 FROM data.document_shares\n WHERE document_id = documents.id\n AND user_id = private.current_user_id()\n AND permission IN ('write', 'admin')\n )\n );\n```\n\n## RLS with SECURITY DEFINER\n\n### Understanding the Interaction\n\n```mermaid\nsequenceDiagram\n participant App as Application\n participant API as api.get_orders()\u003cbr/>SECURITY DEFINER\n participant RLS as RLS Policy\n participant Data as data.orders\n \n App->>API: CALL with user context\n Note over API: Executes as function owner\u003cbr/>(app_owner role)\n API->>API: Set session variables\u003cbr/>(tenant_id, user_id)\n API->>Data: SELECT * FROM data.orders\n Data->>RLS: Check policies\n Note over RLS: Uses current_setting()\u003cbr/>NOT current_user\n RLS-->>Data: Filter rows\n Data-->>API: Return filtered rows\n API-->>App: Return results\n```\n\n### Key Insight: RLS Context\n\nWith `SECURITY DEFINER`:\n- The query runs as the **function owner** (e.g., `app_owner`)\n- RLS policies should use **session variables**, not `current_user`\n- Policies checking `current_user` will see the function owner, not the actual user!\n\n```sql\n-- WRONG: Uses current_user (will be function owner)\nCREATE POLICY wrong_policy ON data.orders\n FOR SELECT\n USING (created_by = current_user); -- ❌ Returns function owner!\n\n-- CORRECT: Uses session variable\nCREATE POLICY correct_policy ON data.orders\n FOR SELECT\n USING (created_by = private.current_user_id()); -- ✅ Returns actual user\n```\n\n### Bypassing RLS for Administrative Functions\n\n```sql\n-- Some functions need to bypass RLS (e.g., admin reports)\n-- Use SECURITY DEFINER + function owner has BYPASSRLS\n\n-- Create a special role with BYPASSRLS\nCREATE ROLE app_admin NOLOGIN BYPASSRLS;\nGRANT app_admin TO app_owner;\n\n-- Admin function that bypasses RLS\nCREATE FUNCTION api.admin_get_all_orders(in_tenant_id uuid)\nRETURNS TABLE (id uuid, customer_name text, total numeric)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nSET ROLE = app_admin -- Switch to role with BYPASSRLS\nAS $\nBEGIN\n -- Verify caller is admin\n IF NOT private.is_current_user_admin() THEN\n RAISE EXCEPTION 'Unauthorized' USING ERRCODE = 'P0050';\n END IF;\n \n RETURN QUERY\n SELECT o.id, c.name, o.total\n FROM data.orders o\n JOIN data.customers c ON c.id = o.customer_id\n WHERE o.tenant_id = in_tenant_id; -- Manual filter, RLS bypassed\nEND;\n$;\n```\n\n## Policy Patterns\n\n### Soft Delete with RLS\n\n```sql\n-- Add deleted flag\nALTER TABLE data.customers ADD COLUMN deleted_at timestamptz;\n\n-- Restrictive policy hides deleted rows\nCREATE POLICY hide_deleted ON data.customers\n AS RESTRICTIVE\n FOR SELECT\n USING (deleted_at IS NULL);\n\n-- Soft delete procedure\nCREATE PROCEDURE api.delete_customer(in_customer_id uuid)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n UPDATE data.customers\n SET deleted_at = now()\n WHERE id = in_customer_id\n AND tenant_id = private.current_tenant_id(); -- RLS would also enforce this\nEND;\n$;\n\n-- Admin function to see deleted records\nCREATE FUNCTION api.admin_list_deleted_customers()\nRETURNS TABLE (id uuid, email text, deleted_at timestamptz)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nSET ROLE = app_admin\nAS $\nBEGIN\n -- Bypasses RLS, can see deleted_at IS NOT NULL\n RETURN QUERY\n SELECT c.id, c.email, c.deleted_at\n FROM data.customers c\n WHERE c.tenant_id = private.current_tenant_id()\n AND c.deleted_at IS NOT NULL;\nEND;\n$;\n```\n\n### Time-Based Access\n\n```sql\n-- Documents with expiration\nCREATE TABLE data.shared_links (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n document_id uuid NOT NULL REFERENCES data.documents(id),\n token text NOT NULL UNIQUE,\n expires_at timestamptz NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Policy: only non-expired links\nCREATE POLICY valid_links ON data.shared_links\n FOR SELECT\n USING (expires_at > now());\n\n-- Access via link token\nCREATE FUNCTION api.get_document_by_link(in_token text)\nRETURNS TABLE (id uuid, title text, content text)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT d.id, d.title, d.content\n FROM data.documents d\n JOIN data.shared_links sl ON sl.document_id = d.id\n WHERE sl.token = in_token;\n -- RLS on shared_links filters expired automatically\n$;\n```\n\n### IP-Based Restrictions\n\n```sql\n-- Store allowed IPs per tenant\nCREATE TABLE data.tenant_allowed_ips (\n tenant_id uuid NOT NULL REFERENCES data.tenants(id),\n ip_range inet NOT NULL,\n PRIMARY KEY (tenant_id, ip_range)\n);\n\n-- Function to check IP\nCREATE OR REPLACE FUNCTION private.is_ip_allowed()\nRETURNS boolean\nLANGUAGE sql\nSTABLE\nAS $\n SELECT EXISTS (\n SELECT 1 FROM data.tenant_allowed_ips\n WHERE tenant_id = private.current_tenant_id()\n AND inet_client_addr() \u003c\u003c= ip_range\n ) OR NOT EXISTS (\n -- If no IPs configured, allow all\n SELECT 1 FROM data.tenant_allowed_ips\n WHERE tenant_id = private.current_tenant_id()\n );\n$;\n\n-- Restrictive policy for IP check\nCREATE POLICY ip_restriction ON data.sensitive_data\n AS RESTRICTIVE\n FOR ALL\n USING (private.is_ip_allowed());\n```\n\n## Performance Considerations\n\n### Index for Policy Conditions\n\n```sql\n-- Always index columns used in RLS policies\nCREATE INDEX orders_tenant_idx ON data.orders(tenant_id);\nCREATE INDEX orders_created_by_idx ON data.orders(created_by);\nCREATE INDEX documents_owner_idx ON data.documents(owner_id);\n\n-- Composite index for common policy patterns\nCREATE INDEX orders_tenant_created_idx ON data.orders(tenant_id, created_by);\n```\n\n### Cache Function Results with Subselect\n\nWhen a policy calls a function (e.g., to get the current user or tenant), PostgreSQL may evaluate it **once per row**. Wrapping the call in a scalar subselect causes the planner to evaluate it once and cache the result.\n\n```sql\n-- BAD: Function called for every row in the table\nCREATE POLICY tenant_isolation ON data.orders\n FOR ALL\n USING (tenant_id = private.current_tenant_id());\n-- On a 1M-row table, private.current_tenant_id() is called 1M times\n\n-- GOOD: Subselect evaluated once, result cached\nCREATE POLICY tenant_isolation ON data.orders\n FOR ALL\n USING (tenant_id = (SELECT private.current_tenant_id()));\n-- 100x+ faster on large tables\n```\n\nApply the same pattern to any function used in a policy expression:\n\n```sql\n-- BAD\nCREATE POLICY user_data ON data.documents\n FOR SELECT\n USING (owner_id = private.current_user_id());\n\n-- GOOD\nCREATE POLICY user_data ON data.documents\n FOR SELECT\n USING (owner_id = (SELECT private.current_user_id()));\n```\n\nFor complex access checks, combine the subselect pattern with a `SECURITY DEFINER` helper function that performs an indexed lookup instead of a per-row check:\n\n```sql\n-- Helper function: runs as definer, bypasses RLS, does indexed lookup\nCREATE FUNCTION private.is_team_member(in_team_id uuid)\nRETURNS boolean\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT EXISTS (\n SELECT 1 FROM data.team_members\n WHERE team_id = in_team_id\n AND user_id = (SELECT private.current_user_id())\n );\n$;\n\n-- Policy uses subselect-wrapped helper\nCREATE POLICY team_orders ON data.orders\n FOR SELECT\n USING ((SELECT private.is_team_member(team_id)));\n```\n\n### Avoid Expensive Policy Functions\n\n```sql\n-- BAD: Subquery in every row check\nCREATE POLICY bad_policy ON data.documents\n FOR SELECT\n USING (\n owner_id IN (\n SELECT user_id FROM data.team_members\n WHERE team_id IN (\n SELECT team_id FROM data.team_members\n WHERE user_id = private.current_user_id()\n )\n )\n );\n\n-- GOOD: Use materialized/cached data\nCREATE TABLE data.user_accessible_teams (\n user_id uuid NOT NULL,\n accessible_user_id uuid NOT NULL,\n PRIMARY KEY (user_id, accessible_user_id)\n);\n\n-- Refresh periodically\nCREATE OR REPLACE FUNCTION private.refresh_team_access()\nRETURNS void\nLANGUAGE sql\nAS $\n TRUNCATE data.user_accessible_teams;\n INSERT INTO data.user_accessible_teams\n SELECT DISTINCT \n tm1.user_id,\n tm2.user_id AS accessible_user_id\n FROM data.team_members tm1\n JOIN data.team_members tm2 ON tm1.team_id = tm2.team_id;\n$;\n\n-- Fast policy\nCREATE POLICY good_policy ON data.documents\n FOR SELECT\n USING (\n owner_id IN (\n SELECT accessible_user_id \n FROM data.user_accessible_teams\n WHERE user_id = private.current_user_id()\n )\n );\n```\n\n### Policy Performance Testing\n\n```sql\n-- Compare query plans with and without RLS\nSET row_security = off;\nEXPLAIN ANALYZE SELECT * FROM data.orders WHERE id = 'uuid';\n\nSET row_security = on;\nEXPLAIN ANALYZE SELECT * FROM data.orders WHERE id = 'uuid';\n\n-- Check if policy adds expensive operations\n```\n\n## Testing RLS\n\n### Test Helper Functions\n\n```sql\nCREATE OR REPLACE FUNCTION test.set_test_context(\n in_tenant_id uuid,\n in_user_id uuid\n)\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n PERFORM set_config('app.current_tenant_id', in_tenant_id::text, true);\n PERFORM set_config('app.current_user_id', in_user_id::text, true);\nEND;\n$;\n\nCREATE OR REPLACE FUNCTION test.test_rls_tenant_isolation()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_tenant1_id uuid;\n l_tenant2_id uuid;\n l_customer_id uuid;\n l_count integer;\nBEGIN\n -- Create two tenants\n INSERT INTO data.tenants (name) VALUES ('Tenant 1') RETURNING id INTO l_tenant1_id;\n INSERT INTO data.tenants (name) VALUES ('Tenant 2') RETURNING id INTO l_tenant2_id;\n \n -- Create customer in tenant 1\n PERFORM test.set_test_context(l_tenant1_id, gen_random_uuid());\n INSERT INTO data.customers (tenant_id, email, name)\n VALUES (l_tenant1_id, '[email protected]', 'Tenant 1 Customer')\n RETURNING id INTO l_customer_id;\n \n -- Verify tenant 1 can see it\n SELECT COUNT(*) INTO l_count FROM data.customers WHERE id = l_customer_id;\n RETURN NEXT is(l_count, 1, 'Tenant 1 should see own customer');\n \n -- Switch to tenant 2\n PERFORM test.set_test_context(l_tenant2_id, gen_random_uuid());\n \n -- Verify tenant 2 cannot see it\n SELECT COUNT(*) INTO l_count FROM data.customers WHERE id = l_customer_id;\n RETURN NEXT is(l_count, 0, 'Tenant 2 should NOT see tenant 1 customer');\n \nEND;\n$;\n```\n\n## Common Pitfalls\n\n### 1. Forgetting to Enable RLS\n\n```sql\n-- Table without RLS is visible to all!\nCREATE TABLE data.secrets (\n id uuid PRIMARY KEY,\n value text\n);\n-- Missing: ALTER TABLE data.secrets ENABLE ROW LEVEL SECURITY;\n\n-- Check all tables have RLS enabled\nSELECT \n schemaname || '.' || tablename AS table_name,\n CASE WHEN rowsecurity THEN '✅ Enabled' ELSE '❌ DISABLED' END AS rls_status\nFROM pg_tables\nWHERE schemaname = 'data'\nORDER BY rowsecurity, tablename;\n```\n\n### 2. Policy Allows Nothing (Locked Out)\n\n```sql\n-- If no policy matches, access is denied!\nALTER TABLE data.orders ENABLE ROW LEVEL SECURITY;\n-- No policies created = no one can access\n\n-- Always create at least one policy\nCREATE POLICY default_deny ON data.orders\n FOR ALL\n USING (false); -- Explicit deny\n\nCREATE POLICY tenant_access ON data.orders\n FOR ALL\n USING (tenant_id = private.current_tenant_id());\n```\n\n### 3. UPDATE/DELETE Without Proper Policy\n\n```sql\n-- SELECT policy doesn't cover UPDATE/DELETE\nCREATE POLICY read_own ON data.documents\n FOR SELECT\n USING (owner_id = private.current_user_id());\n\n-- Need separate policies for modifications\nCREATE POLICY write_own ON data.documents\n FOR UPDATE\n USING (owner_id = private.current_user_id())\n WITH CHECK (owner_id = private.current_user_id());\n\nCREATE POLICY delete_own ON data.documents\n FOR DELETE\n USING (owner_id = private.current_user_id());\n```\n\n### 4. INSERT Without WITH CHECK\n\n```sql\n-- USING only checks existing rows\n-- Need WITH CHECK for new rows\nCREATE POLICY insert_own ON data.documents\n FOR INSERT\n WITH CHECK (\n owner_id = private.current_user_id()\n AND tenant_id = private.current_tenant_id()\n );\n```\n\n### 5. Leaking Data Through Functions\n\n```sql\n-- BAD: Function exposes row count even for inaccessible rows\nCREATE FUNCTION api.count_all_orders()\nRETURNS bigint\nLANGUAGE sql\nSECURITY DEFINER\nSET ROLE = app_admin -- Bypasses RLS!\nAS $\n SELECT COUNT(*) FROM data.orders;\n$;\n\n-- GOOD: Respect RLS\nCREATE FUNCTION api.count_my_orders()\nRETURNS bigint\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT COUNT(*) FROM data.orders; -- RLS filters automatically\n$;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":21622,"content_sha256":"6f955c0829157b680420b734226c0f865f80d19ff8e9f33adc2782c51b23f50d"},{"filename":"references/schema-architecture.md","content":"# Schema Architecture Best Practices\n\n## Philosophy\n\nPostgreSQL schemas provide namespaces for organizing database objects. Best practices recommend separating concerns into distinct schemas rather than placing everything in the `public` schema.\n\n**Core Principle**: Use schema separation to:\n1. Isolate internal implementation from external interfaces\n2. Control access at the schema level\n3. Enable API versioning without breaking changes\n4. Protect sensitive data and internal logic\n\n## Recommended Schema Structure\n\n### High-Level Architecture\n\n```mermaid\nflowchart TB\n subgraph APP[\"Application Layer\"]\n Client[\"Client Application\"]\n end\n \n subgraph DB[\"PostgreSQL Database\"]\n subgraph PUBLIC_API[\"Public Interface\"]\n API[\"api schema\u003cbr/>━━━━━━━━━━━━━\u003cbr/>Functions\u003cbr/>Procedures\u003cbr/>Views\"]\n end\n \n subgraph INTERNAL[\"Internal Implementation\"]\n DATA[\"data schema\u003cbr/>━━━━━━━━━━━━━\u003cbr/>Tables\u003cbr/>Indexes\u003cbr/>Constraints\"]\n PRIVATE[\"private schema\u003cbr/>━━━━━━━━━━━━━\u003cbr/>Triggers\u003cbr/>Internal Functions\u003cbr/>Helpers\"]\n end\n \n subgraph SUPPORT[\"Support Schemas\"]\n AUDIT[\"app_audit\u003cbr/>━━━━━━━━━━━━━\u003cbr/>Audit Logs\"]\n MIG[\"app_migration\u003cbr/>━━━━━━━━━━━━━\u003cbr/>Migration Tracking\"]\n end\n end\n \n Client -->|\"EXECUTE permission\"| API\n API -->|\"SECURITY DEFINER\"| DATA\n API -->|\"Calls internal functions\"| PRIVATE\n PRIVATE -->|\"Triggers fire on\"| DATA\n PRIVATE -.->|\"Logs changes\"| AUDIT\n \n style APP fill:#e3f2fd\n style PUBLIC_API fill:#c8e6c9\n style INTERNAL fill:#fff3e0\n style SUPPORT fill:#f3e5f5\n```\n\n### Schema Dependency Graph\n\n```mermaid\ngraph LR\n subgraph \"External Access\"\n APP[Application]\n end\n \n subgraph \"Database Schemas\"\n API[api]\n DATA[data]\n PRIVATE[private]\n AUDIT[app_audit]\n MIG[app_migration]\n end\n \n APP -->|\"can call\"| API\n API -->|\"reads/writes\"| DATA\n API -->|\"uses\"| PRIVATE\n PRIVATE -->|\"triggers on\"| DATA\n PRIVATE -->|\"writes to\"| AUDIT\n \n APP x-.->|\"blocked\"| DATA\n APP x-.->|\"blocked\"| PRIVATE\n APP x-.->|\"blocked\"| AUDIT\n \n style API fill:#c8e6c9,stroke:#2e7d32\n style DATA fill:#ffecb3,stroke:#f57f17\n style PRIVATE fill:#ffe0b2,stroke:#e65100\n style AUDIT fill:#e1bee7,stroke:#7b1fa2\n style MIG fill:#e1bee7,stroke:#7b1fa2\n```\n\n### Permission Model\n\n```mermaid\nflowchart TD\n subgraph ROLES[\"Database Roles\"]\n OWNER[\"app_owner\u003cbr/>(NOLOGIN)\"]\n READ[\"app_read\"]\n WRITE[\"app_write\"]\n ADMIN[\"app_admin\"]\n SERVICE[\"app_service\u003cbr/>(LOGIN)\"]\n end\n \n subgraph SCHEMAS[\"Schema Permissions\"]\n S_DATA[\"data schema\"]\n S_PRIVATE[\"private schema\"]\n S_API[\"api schema\"]\n S_AUDIT[\"app_audit schema\"]\n end\n \n OWNER -->|\"owns all objects\"| S_DATA\n OWNER -->|\"owns all objects\"| S_PRIVATE\n OWNER -->|\"owns all objects\"| S_API\n \n READ -->|\"EXECUTE functions\"| S_API\n WRITE -->|\"EXECUTE procedures\"| S_API\n ADMIN -->|\"SELECT\"| S_AUDIT\n \n SERVICE -->|\"inherits\"| WRITE\n WRITE -->|\"inherits\"| READ\n ADMIN -->|\"inherits\"| WRITE\n \n style OWNER fill:#ffcdd2\n style SERVICE fill:#c8e6c9\n style S_API fill:#c8e6c9\n style S_DATA fill:#fff3e0\n style S_PRIVATE fill:#fff3e0\n```\n\n## Schema Definitions\n\n### 1. `data` Schema - Raw Tables\n\nThe `data` schema contains all base tables. No functions, no views—just tables and their constraints.\n\n```sql\nCREATE SCHEMA data;\nCOMMENT ON SCHEMA data IS 'Base tables - no direct external access';\n\n-- Tables only\nCREATE TABLE data.customers (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n email text NOT NULL,\n name text NOT NULL,\n password_hash text NOT NULL, -- Sensitive!\n is_active boolean NOT NULL DEFAULT true,\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE TABLE data.orders (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n customer_id uuid NOT NULL REFERENCES data.customers(id),\n status text NOT NULL DEFAULT 'pending',\n total numeric(15,2) NOT NULL,\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Indexes\nCREATE UNIQUE INDEX customers_email_key ON data.customers(lower(email));\nCREATE INDEX orders_customer_id_idx ON data.orders(customer_id);\nCREATE INDEX orders_status_idx ON data.orders(status) WHERE status NOT IN ('completed', 'cancelled');\n```\n\n**Rules for `data` schema:**\n- Contains ONLY tables, indexes, and constraints\n- No functions, procedures, triggers, or views\n- No direct grants to application roles\n- Accessed only through `api` schema functions (which use SECURITY DEFINER)\n\n### 2. `private` Schema - Internal Logic\n\nThe `private` schema contains:\n- Trigger functions\n- Internal business logic functions\n- Helper functions\n- Anything the application shouldn't call directly\n\n```sql\nCREATE SCHEMA private;\nCOMMENT ON SCHEMA private IS 'Internal functions and triggers - not exposed externally';\n\n-- Trigger function (internal)\nCREATE OR REPLACE FUNCTION private.set_updated_at()\nRETURNS trigger\nLANGUAGE plpgsql\nAS $\nBEGIN\n NEW.updated_at := now();\n RETURN NEW;\nEND;\n$;\n\n-- Apply triggers\nCREATE TRIGGER customers_bu_updated_trg\n BEFORE UPDATE ON data.customers\n FOR EACH ROW\n EXECUTE FUNCTION private.set_updated_at();\n\nCREATE TRIGGER orders_bu_updated_trg\n BEFORE UPDATE ON data.orders\n FOR EACH ROW\n EXECUTE FUNCTION private.set_updated_at();\n\n-- Internal helper function\nCREATE OR REPLACE FUNCTION private.hash_password(in_password text)\nRETURNS text\nLANGUAGE sql\nIMMUTABLE\nPARALLEL SAFE\nAS $\n SELECT crypt(in_password, gen_salt('bf', 10));\n$;\n\n-- Internal validation\nCREATE OR REPLACE FUNCTION private.verify_password(\n in_password text,\n in_hash text\n)\nRETURNS boolean\nLANGUAGE sql\nIMMUTABLE\nPARALLEL SAFE\nAS $\n SELECT in_hash = crypt(in_password, in_hash);\n$;\n\n-- Internal audit logging\nCREATE OR REPLACE FUNCTION private.log_audit(\n in_table_name text,\n in_operation text,\n in_old_data jsonb,\n in_new_data jsonb\n)\nRETURNS void\nLANGUAGE sql\nAS $\n INSERT INTO app_audit.changelog (table_name, operation, old_data, new_data)\n VALUES (in_table_name, in_operation, in_old_data, in_new_data);\n$;\n```\n\n**Rules for `private` schema:**\n- Contains internal logic only\n- Not exposed to API or external access\n- Houses trigger functions\n- Contains helper functions used by `api` schema\n- May access `data` schema directly\n\n### 3. `api` Schema - External Interface (Table API)\n\nThe `api` schema is the **only** schema exposed to applications. It contains:\n- Functions and procedures (Table API)\n- Views (for read-only access where appropriate)\n- No direct table access\n\n```sql\nCREATE SCHEMA api;\nCOMMENT ON SCHEMA api IS 'Public API - external interface for applications';\n\n-- ============================================================================\n-- CUSTOMER API\n-- ============================================================================\n\n-- Read: Get customer by ID (excludes sensitive fields)\nCREATE OR REPLACE FUNCTION api.get_customer(in_id uuid)\nRETURNS TABLE (\n id uuid,\n email text,\n name text,\n is_active boolean,\n created_at timestamptz,\n updated_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name, is_active, created_at, updated_at\n FROM data.customers\n WHERE id = in_id;\n$;\n\n-- Read: List customers with pagination\nCREATE OR REPLACE FUNCTION api.select_customers(\n in_is_active boolean DEFAULT NULL,\n in_limit integer DEFAULT 100,\n in_offset integer DEFAULT 0\n)\nRETURNS TABLE (\n id uuid,\n email text,\n name text,\n is_active boolean,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, email, name, is_active, created_at\n FROM data.customers\n WHERE in_is_active IS NULL OR is_active = in_is_active\n ORDER BY created_at DESC\n LIMIT in_limit OFFSET in_offset;\n$;\n\n-- Write: Insert customer\nCREATE OR REPLACE PROCEDURE api.insert_customer(\n in_email text,\n in_name text,\n in_password text,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.customers (email, name, password_hash)\n VALUES (\n lower(trim(in_email)), \n trim(in_name), \n private.hash_password(in_password)\n )\n RETURNING id INTO io_id;\nEND;\n$;\n\n-- Write: Update customer\nCREATE OR REPLACE PROCEDURE api.update_customer(\n in_id uuid,\n in_name text DEFAULT NULL,\n in_email text DEFAULT NULL,\n in_is_active boolean DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n UPDATE data.customers\n SET \n name = COALESCE(trim(in_name), name),\n email = COALESCE(lower(trim(in_email)), email),\n is_active = COALESCE(in_is_active, is_active)\n WHERE id = in_id;\n \n IF NOT FOUND THEN\n RAISE EXCEPTION 'Customer not found: %', in_id\n USING ERRCODE = 'P0002'; -- no_data_found\n END IF;\nEND;\n$;\n\n-- Write: Authenticate customer (returns customer data or NULL)\nCREATE OR REPLACE FUNCTION api.authenticate_customer(\n in_email text,\n in_password text\n)\nRETURNS TABLE (\n id uuid,\n email text,\n name text\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT c.id, c.email, c.name\n FROM data.customers c\n WHERE c.email = lower(trim(in_email))\n AND c.is_active = true\n AND private.verify_password(in_password, c.password_hash);\n$;\n\n-- ============================================================================\n-- ORDER API\n-- ============================================================================\n\n-- Read: Get order with customer info\nCREATE OR REPLACE FUNCTION api.get_order(in_id uuid)\nRETURNS TABLE (\n id uuid,\n customer_id uuid,\n customer_email text,\n customer_name text,\n status text,\n total numeric,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT \n o.id, o.customer_id, c.email, c.name,\n o.status, o.total, o.created_at\n FROM data.orders o\n JOIN data.customers c ON c.id = o.customer_id\n WHERE o.id = in_id;\n$;\n\n-- Read: List orders for customer\nCREATE OR REPLACE FUNCTION api.select_orders_by_customer(\n in_customer_id uuid,\n in_status text DEFAULT NULL,\n in_limit integer DEFAULT 100\n)\nRETURNS TABLE (\n id uuid,\n status text,\n total numeric,\n created_at timestamptz\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT id, status, total, created_at\n FROM data.orders\n WHERE customer_id = in_customer_id\n AND (in_status IS NULL OR status = in_status)\n ORDER BY created_at DESC\n LIMIT in_limit;\n$;\n\n-- Write: Create order\nCREATE OR REPLACE PROCEDURE api.insert_order(\n in_customer_id uuid,\n in_total numeric,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n -- Verify customer exists and is active\n IF NOT EXISTS (\n SELECT 1 FROM data.customers \n WHERE id = in_customer_id AND is_active = true\n ) THEN\n RAISE EXCEPTION 'Customer not found or inactive: %', in_customer_id\n USING ERRCODE = 'P0002';\n END IF;\n \n INSERT INTO data.orders (customer_id, total)\n VALUES (in_customer_id, in_total)\n RETURNING id INTO io_id;\nEND;\n$;\n\n-- Write: Update order status\nCREATE OR REPLACE PROCEDURE api.update_order_status(\n in_id uuid,\n in_status text\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_current_status text;\nBEGIN\n SELECT status INTO l_current_status\n FROM data.orders WHERE id = in_id;\n\n IF NOT FOUND THEN\n RAISE EXCEPTION 'Order not found: %', in_id\n USING ERRCODE = 'P0002';\n END IF;\n\n -- Business rule: Can't change completed/cancelled orders\n IF l_current_status IN ('completed', 'cancelled') THEN\n RAISE EXCEPTION 'Cannot modify % order', l_current_status\n USING ERRCODE = 'P0001';\n END IF;\n\n UPDATE data.orders\n SET status = in_status\n WHERE id = in_id;\nEND;\n$;\n```\n\n**Rules for `api` schema:**\n- Contains ONLY functions, procedures, and views\n- No tables\n- All functions use `SECURITY DEFINER` with explicit `search_path`\n- Never exposes sensitive columns (passwords, internal IDs, etc.)\n- All write operations are procedures, reads are functions\n- Validates business rules before data access\n\n### 4. `app_audit` Schema - Audit Logs\n\n```sql\nCREATE SCHEMA app_audit;\nCOMMENT ON SCHEMA app_audit IS 'Audit logging - append-only';\n\nCREATE TABLE app_audit.changelog (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n table_name text NOT NULL,\n operation text NOT NULL,\n old_data jsonb,\n new_data jsonb,\n changed_at timestamptz NOT NULL DEFAULT now(),\n changed_by text NOT NULL DEFAULT current_user\n);\n\nCREATE INDEX changelog_table_idx ON app_audit.changelog(table_name, changed_at DESC);\n```\n\n### 5. `app_migration` Schema - Migrations\n\nSee [migrations.md](migrations.md) for the complete migration system implementation.\n\n## Security Configuration\n\n### Role Setup\n\n```sql\n-- Schema owner (cannot login)\nCREATE ROLE app_owner NOLOGIN;\n\n-- Application roles\nCREATE ROLE app_read; -- Read-only access\nCREATE ROLE app_write; -- Read + write access\nCREATE ROLE app_admin; -- Full access\n\n-- Inheritance\nGRANT app_read TO app_write;\nGRANT app_write TO app_admin;\n```\n\n### Schema Permissions\n\n```sql\n-- Revoke public access to public schema\nREVOKE ALL ON SCHEMA public FROM PUBLIC;\n\n-- data schema: No direct access\nREVOKE ALL ON SCHEMA data FROM PUBLIC;\nGRANT USAGE ON SCHEMA data TO app_owner;\n\n-- private schema: No direct access \nREVOKE ALL ON SCHEMA private FROM PUBLIC;\nGRANT USAGE ON SCHEMA private TO app_owner;\n\n-- api schema: Application access\nGRANT USAGE ON SCHEMA api TO app_read, app_write;\n\n-- Grant execute on all API functions\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA api TO app_read;\nGRANT EXECUTE ON ALL PROCEDURES IN SCHEMA api TO app_write;\n\n-- Set defaults for future objects\nALTER DEFAULT PRIVILEGES IN SCHEMA api \n GRANT EXECUTE ON FUNCTIONS TO app_read;\nALTER DEFAULT PRIVILEGES IN SCHEMA api \n GRANT EXECUTE ON PROCEDURES TO app_write;\n\n-- audit schema: Read for admins only\nGRANT USAGE ON SCHEMA app_audit TO app_admin;\nGRANT SELECT ON ALL TABLES IN SCHEMA app_audit TO app_admin;\n```\n\n### Application Connection Role\n\n```sql\n-- Application connection role\nCREATE ROLE app_service LOGIN PASSWORD 'secure_password';\nGRANT app_write TO app_service;\n\n-- Set search path for application\nALTER ROLE app_service SET search_path = api, pg_temp;\n```\n\n## Search Path Configuration\n\n### For SECURITY DEFINER Functions\n\nAlways set explicit search_path:\n\n```sql\nCREATE FUNCTION api.some_function()\nRETURNS void\nSECURITY DEFINER\nSET search_path = data, private, pg_temp -- Explicit, secure\nAS $ ... $;\n```\n\n### For Application Connections\n\n```sql\n-- In connection string or role configuration\nSET search_path = api, pg_temp;\n```\n\n## API Versioning\n\nWhen you need breaking changes:\n\n```sql\n-- Create new API version\nCREATE SCHEMA api_v2;\n\n-- Copy and modify functions\nCREATE FUNCTION api_v2.get_customer(...) ...;\n\n-- Maintain both versions\nGRANT USAGE ON SCHEMA api_v1 TO app_read;\nGRANT USAGE ON SCHEMA api_v2 TO app_read;\n\n-- Deprecate old version eventually\nCOMMENT ON SCHEMA api_v1 IS 'DEPRECATED: Use api_v2. Will be removed 2025-01-01';\n```\n\n## File Organization\n\nOrganize SQL files by schema:\n\n```\nmigrations/\n├── V001__create_schemas.sql\n├── V002__create_data_tables.sql\n├── V003__create_audit_tables.sql\n├── V004__create_roles.sql\n├── V005__grant_permissions.sql\n└── repeatable/\n ├── R__data_indexes.sql\n ├── R__private_functions.sql\n ├── R__private_triggers.sql\n ├── R__api_customer_functions.sql\n ├── R__api_order_functions.sql\n └── R__api_views.sql\n```\n\n## Quick Reference\n\n| Schema | Contains | Direct Access | Purpose |\n|--------|----------|---------------|---------|\n| `data` | Tables, indexes, constraints | None (app_owner only) | Data storage |\n| `private` | Triggers, internal functions | None (app_owner only) | Internal logic |\n| `api` | Functions, procedures, views | app_read, app_write | External interface |\n| `app_audit` | Audit tables | app_admin (read only) | Audit trail |\n| `app_migration` | Migration tracking | app_admin | Schema versioning |\n\n## Benefits of This Approach\n\n1. **Security**: Sensitive data (passwords, internal IDs) never exposed\n2. **Encapsulation**: Internal changes don't break external interfaces\n3. **Versioning**: API schemas can be versioned independently\n4. **Permissions**: Grant access at schema level, not object level\n5. **Clarity**: Clear separation of concerns\n6. **Auditability**: All changes flow through controlled functions\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":17403,"content_sha256":"eafde77f2b127b5ddaab0840648997fe8234fb305df78103f5afe1c1a32071de"},{"filename":"references/schema-naming.md","content":"# Schema Design & Naming Conventions\n\n## Table of Contents\n1. [Schema Organization](#schema-organization)\n2. [Naming Rules](#naming-rules)\n3. [Table Naming](#table-naming)\n4. [Column Naming](#column-naming)\n5. [Index Naming](#index-naming)\n6. [Constraint Naming](#constraint-naming)\n7. [Function & Procedure Naming](#function--procedure-naming)\n8. [Trigger Naming](#trigger-naming)\n\n## Schema Organization\n\n### Use Named Schemas\n\nCreate a single database with multiple named schemas. Remove the public schema.\n\n```sql\n-- Create application schemas\nCREATE SCHEMA app; -- Main application objects\nCREATE SCHEMA app_audit; -- Audit logging\nCREATE SCHEMA app_migration; -- Migration system\nCREATE SCHEMA app_api; -- External API functions\n\n-- Revoke public access\nREVOKE ALL ON SCHEMA public FROM PUBLIC;\n\n-- Grant schema access to roles\nGRANT USAGE ON SCHEMA app TO app_role;\nGRANT USAGE ON SCHEMA app_api TO api_role;\n```\n\n### Schema Benefits\n- Namespace isolation prevents naming conflicts\n- Simplified permission management at schema level\n- Logical grouping of related objects\n- Multi-tenant support via schema-per-tenant\n\n## Naming Rules\n\n### Universal Rules\n\n1. **Lowercase only**: Never use uppercase. PostgreSQL folds unquoted identifiers to lowercase; mixed case requires double-quoting everywhere and causes tool incompatibility.\n\n2. **Snake_case**: Use underscores to separate words: `order_items`, not `orderItems` or `OrderItems`.\n\n3. **No reserved words**: Avoid SQL reserved words as identifiers.\n\n4. **No prefixes**: Don't use `tbl_`, `sp_`, `fn_` prefixes. Don't prefix with `pg_` (reserved).\n\n5. **Maximum 63 characters**: PostgreSQL truncates longer names.\n\n6. **Descriptive names**: Prefer clarity over brevity. `customer_shipping_address` over `cust_ship_addr`.\n\n### Prohibited Characters\n- Spaces\n- Special characters except underscore\n- Leading numbers\n\n## Table Naming\n\n### Rules\n- Use **plural nouns**: `orders`, `customers`, `order_items`\n- Use **snake_case**: `order_line_items`\n- No prefixes: `orders`, not `tbl_orders`\n\n### Special Table Prefixes\n| Prefix | Use |\n|--------|-----|\n| `v_` | Views |\n| `mv_` | Materialized views |\n| `tmp_` | Temporary tables |\n\n### Join Table Naming\nFor many-to-many relationships, combine both table names alphabetically:\n```sql\n-- users \u003c-> roles many-to-many\nCREATE TABLE data.roles_users (\n role_id uuid REFERENCES data.roles(id),\n user_id uuid REFERENCES data.users(id),\n PRIMARY KEY (role_id, user_id)\n);\n```\n\n## Column Naming\n\n### Primary Keys\n```sql\n-- Option 1: Simple 'id' (preferred for most cases)\nid uuid PRIMARY KEY DEFAULT uuidv7()\n\n-- Option 2: Table-prefixed (for clarity in complex joins)\norder_id uuid PRIMARY KEY DEFAULT uuidv7()\n```\n\n### Foreign Keys\nName as `{referenced_table_singular}_id`:\n```sql\ncustomer_id uuid REFERENCES data.customers(id)\nparent_order_id uuid REFERENCES data.orders(id) -- self-reference\n```\n\n### Timestamp Columns\n```sql\ncreated_at timestamptz NOT NULL DEFAULT now()\nupdated_at timestamptz NOT NULL DEFAULT now()\ndeleted_at timestamptz -- for soft deletes\n```\n\n### Boolean Columns\nUse `is_` or `has_` prefix:\n```sql\nis_active boolean NOT NULL DEFAULT true\nis_verified boolean NOT NULL DEFAULT false\nhas_subscription boolean NOT NULL DEFAULT false\n```\n\n### Status/State Columns\n```sql\nstatus text NOT NULL DEFAULT 'pending'\norder_state text NOT NULL DEFAULT 'draft'\n```\n\n### Numeric Columns\nUse descriptive suffixes:\n```sql\nquantity integer\ntotal_amount numeric(15,2)\ndiscount_rate numeric(5,4)\nretry_count integer DEFAULT 0\n```\n\n## Index Naming\n\n### Standard Indexes\nPattern: `{table}_{column(s)}_idx` (Trivadis v4.4)\n```sql\nCREATE INDEX orders_customer_id_idx ON data.orders(customer_id);\nCREATE INDEX orders_status_created_idx ON data.orders(status, created_at DESC);\n```\n\n### Unique Indexes\nPattern: `{table}_{column(s)}_key`\n```sql\nCREATE UNIQUE INDEX users_email_key ON data.users(lower(email));\n```\n\n### Partial Indexes\nInclude condition hint:\n```sql\nCREATE INDEX orders_pending_idx ON data.orders(created_at)\n WHERE status = 'pending';\n```\n\n### Expression Indexes\n```sql\nCREATE INDEX users_email_lower_idx ON data.users(lower(email));\n```\n\n## Constraint Naming\n\n> **Note**: PostgreSQL auto-generates constraint names with suffixes like `_pkey`, `_fkey`, `_key`, `_check`. The patterns below use shorter Trivadis-style suffixes for explicit naming. Both approaches are acceptable; explicit naming provides clearer error messages.\n\n### Primary Keys\nPattern: `{table}_pk`\n```sql\nALTER TABLE data.orders ADD CONSTRAINT orders_pk PRIMARY KEY (id);\n```\n\n### Foreign Keys\nPattern: `{table}_{reftable}_fk`\n```sql\nALTER TABLE data.orders\n ADD CONSTRAINT orders_customers_fk\n FOREIGN KEY (customer_id) REFERENCES data.customers(id);\n```\n\n### Unique Constraints\nPattern: `{table}_{column(s)}_uk`\n```sql\nALTER TABLE data.users\n ADD CONSTRAINT users_email_uk UNIQUE (email);\n```\n\n### Check Constraints\nPattern: `{table}_{column}_ck`\n```sql\nALTER TABLE data.orders\n ADD CONSTRAINT orders_status_ck\n CHECK (status IN ('draft', 'pending', 'confirmed', 'shipped', 'delivered', 'cancelled'));\n\nALTER TABLE data.orders\n ADD CONSTRAINT orders_total_ck\n CHECK (total >= 0);\n```\n\n### Exclusion Constraints\nPattern: `{table}_{description}_excl`\n```sql\nALTER TABLE data.reservations \n ADD CONSTRAINT reservations_no_overlap_excl \n EXCLUDE USING gist (room_id WITH =, during WITH &&);\n```\n\n## Function & Procedure Naming\n\n### Action Prefixes\n| Prefix | Returns | Use |\n|--------|---------|-----|\n| `select_` | SETOF/TABLE | Query operations |\n| `get_` | Single value/row | Fetch one item |\n| `insert_` | void/id | Create new record |\n| `update_` | void/count | Modify existing |\n| `delete_` | void/count | Remove record |\n| `upsert_` | void/id | Insert or update |\n| `validate_` | boolean | Check validity |\n| `calculate_` | value | Compute result |\n\n### Naming Pattern\n`{action}_{entity}[_by{filter}][_with{modifier}]`\n\n```sql\n-- Query functions\nselect_orders()\nselect_orders_by_customer(in_customer_id)\nselect_orders_by_status_and_date(in_status, in_start_date, in_end_date)\nget_order_by_id(in_order_id)\nget_customer_balance(in_customer_id)\n\n-- Mutation procedures\ninsert_order(in_customer_id, in_items)\nupdate_order_status(in_order_id, in_new_status)\ndelete_order(in_order_id)\nupsert_customer(in_email, in_name)\n```\n\n### Parameter Naming\nPrefix all parameters with `in_` to avoid conflicts with column names:\n```sql\nCREATE FUNCTION api.select_orders_by_customer(\n in_customer_id uuid,\n in_limit integer DEFAULT 100,\n in_offset integer DEFAULT 0\n)\n```\n\n## Trigger Naming\n\n### Pattern\n`{table}_{timing}{event(s)}_trg`\n\nWhere:\n- Timing: `b` (before), `a` (after), `i` (instead of)\n- Events: `i` (insert), `u` (update), `d` (delete), `t` (truncate)\n\n### Examples\n```sql\n-- Before insert/update trigger (trigger function in private schema)\nCREATE TRIGGER orders_biu_trg\n BEFORE INSERT OR UPDATE ON data.orders\n FOR EACH ROW EXECUTE FUNCTION private.set_updated_at();\n\n-- After insert trigger (audit function in private schema)\nCREATE TRIGGER orders_ai_trg\n AFTER INSERT ON data.orders\n FOR EACH ROW EXECUTE FUNCTION private.log_audit();\n\n-- After delete trigger\nCREATE TRIGGER orders_ad_trg\n AFTER DELETE ON data.orders\n FOR EACH ROW EXECUTE FUNCTION private.log_audit();\n```\n\n### Trigger Function Naming\nPattern: `{action}` or `{table}_{action}` - placed in `private` schema\n```sql\nCREATE FUNCTION private.set_updated_at() RETURNS trigger ...\nCREATE FUNCTION private.log_audit() RETURNS trigger ...\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":7611,"content_sha256":"8e22fa519eb93b5d9a9ec8c83423fda2066ebe7fcc33d91ab3aec113e7c97380"},{"filename":"references/session-management.md","content":"# Connection and Session Management\n\nThis document covers session variables, application context, connection pooling integration, and session lifecycle management.\n\n## Table of Contents\n\n1. [Session Variables](#session-variables)\n2. [Application Context Pattern](#application-context-pattern)\n3. [Connection Pooling Integration](#connection-pooling-integration)\n4. [Session Configuration](#session-configuration)\n5. [Connection Lifecycle](#connection-lifecycle)\n6. [Security Context](#security-context)\n\n## Session Variables\n\n### Setting Session Variables\n\n```sql\n-- Set variable for entire session (persists until connection closes)\nSET myapp.current_user_id = 'user-uuid-here';\nSET myapp.current_tenant_id = 'tenant-uuid-here';\n\n-- Set variable for current transaction only\nSET LOCAL myapp.current_user_id = 'user-uuid-here';\n\n-- Using set_config function (more flexible)\nSELECT set_config('myapp.current_user_id', 'user-uuid-here', false); -- session\nSELECT set_config('myapp.current_user_id', 'user-uuid-here', true); -- transaction only\n```\n\n### Reading Session Variables\n\n```sql\n-- Get variable (returns NULL if not set)\nSELECT current_setting('myapp.current_user_id', true); -- true = missing_ok\n\n-- Get variable (throws error if not set)\nSELECT current_setting('myapp.current_user_id');\n\n-- In PL/pgSQL functions\nCREATE OR REPLACE FUNCTION private.get_current_user_id()\nRETURNS uuid\nLANGUAGE sql\nSTABLE\nAS $\n SELECT NULLIF(current_setting('myapp.current_user_id', true), '')::uuid;\n$;\n\nCREATE OR REPLACE FUNCTION private.get_current_tenant_id()\nRETURNS uuid\nLANGUAGE sql\nSTABLE\nAS $\n SELECT NULLIF(current_setting('myapp.current_tenant_id', true), '')::uuid;\n$;\n```\n\n### Custom GUC Variables\n\n```sql\n-- Register custom variable namespace in postgresql.conf:\n-- custom_variable_classes = 'myapp'\n\n-- Or dynamically (requires superuser)\nALTER SYSTEM SET custom_variable_classes = 'myapp';\nSELECT pg_reload_conf();\n\n-- Now you can use typed variables\nSET myapp.debug_mode = 'true';\nSET myapp.log_level = 'debug';\nSET myapp.max_results = '100';\n```\n\n## Application Context Pattern\n\n### Context Setup Function\n\n```sql\n-- Comprehensive context setup\nCREATE OR REPLACE FUNCTION api.set_context(\n in_user_id uuid,\n in_tenant_id uuid DEFAULT NULL,\n in_role text DEFAULT 'user',\n in_session_id text DEFAULT NULL,\n in_metadata jsonb DEFAULT '{}'\n)\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Core context\n PERFORM set_config('myapp.current_user_id', in_user_id::text, false);\n PERFORM set_config('myapp.current_tenant_id', COALESCE(in_tenant_id::text, ''), false);\n PERFORM set_config('myapp.current_role', in_role, false);\n PERFORM set_config('myapp.session_id', COALESCE(in_session_id, ''), false);\n \n -- Additional metadata\n PERFORM set_config('myapp.context_metadata', in_metadata::text, false);\n \n -- Set timestamp for context creation\n PERFORM set_config('myapp.context_created_at', now()::text, false);\n \n -- Set application name for pg_stat_activity\n PERFORM set_config('application_name', \n format('myapp[user=%s,tenant=%s]', in_user_id, in_tenant_id), \n false);\nEND;\n$;\n\n-- Transaction-scoped context (for pooled connections)\nCREATE OR REPLACE FUNCTION api.set_transaction_context(\n in_user_id uuid,\n in_tenant_id uuid DEFAULT NULL\n)\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Use LOCAL for transaction scope\n PERFORM set_config('myapp.current_user_id', in_user_id::text, true);\n PERFORM set_config('myapp.current_tenant_id', COALESCE(in_tenant_id::text, ''), true);\nEND;\n$;\n```\n\n### Context Validation\n\n```sql\n-- Require context to be set\nCREATE OR REPLACE FUNCTION private.require_context()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_user_id uuid;\nBEGIN\n l_user_id := private.get_current_user_id();\n \n IF l_user_id IS NULL THEN\n RAISE EXCEPTION 'User context not set. Call api.set_context() first.'\n USING ERRCODE = 'P0050'; -- Custom error code\n END IF;\nEND;\n$;\n\n-- API functions that require context\nCREATE OR REPLACE FUNCTION api.get_my_profile()\nRETURNS TABLE (id uuid, email text, name text)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_user_id uuid;\nBEGIN\n -- Require context\n PERFORM private.require_context();\n l_user_id := private.get_current_user_id();\n \n RETURN QUERY\n SELECT u.id, u.email, u.name\n FROM data.users u\n WHERE u.id = l_user_id;\nEND;\n$;\n```\n\n### Context for Auditing\n\n```sql\n-- Set audit context\nCREATE OR REPLACE FUNCTION api.set_audit_context(\n in_reason text DEFAULT NULL,\n in_ticket text DEFAULT NULL\n)\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n PERFORM set_config('myapp.audit_context', \n jsonb_build_object(\n 'reason', in_reason,\n 'ticket', in_ticket,\n 'timestamp', now()\n )::text, \n true); -- Transaction-local\nEND;\n$;\n\n-- Usage in audit trigger\nCREATE OR REPLACE FUNCTION private.get_audit_context()\nRETURNS jsonb\nLANGUAGE sql\nSTABLE\nAS $\n SELECT NULLIF(current_setting('myapp.audit_context', true), '')::jsonb;\n$;\n```\n\n## Connection Pooling Integration\n\n### PgBouncer Configuration\n\n```ini\n# pgbouncer.ini\n[databases]\nmyapp = host=localhost dbname=myapp\n\n[pgbouncer]\npool_mode = transaction # Best for web apps with SECURITY DEFINER\nmax_client_conn = 1000\ndefault_pool_size = 20\nmin_pool_size = 5\nreserve_pool_size = 5\nreserve_pool_timeout = 3\n\n# Important for session variables\nserver_reset_query = DISCARD ALL\n```\n\n### Transaction-Mode Pooling Pattern\n\n```sql\n-- Application must set context at start of each transaction\n-- because connections are shared\n\n-- Example application flow:\nBEGIN;\nSELECT api.set_transaction_context('user-id'::uuid, 'tenant-id'::uuid);\n\n-- Now execute queries\nSELECT * FROM api.get_my_orders();\nSELECT * FROM api.get_my_profile();\n\nCOMMIT;\n-- Connection returns to pool, context is discarded\n```\n\n### Session-Mode Pooling Pattern\n\n```sql\n-- If using session pooling, set context once after connect\n-- Connection stays dedicated to this client\n\nSELECT api.set_context(\n in_user_id := 'user-id'::uuid,\n in_tenant_id := 'tenant-id'::uuid,\n in_session_id := 'app-session-123'\n);\n\n-- All subsequent queries use this context\nSELECT * FROM api.get_my_orders();\n-- ... many more queries ...\n\n-- Context persists until disconnect\n```\n\n### Connection Validation\n\n```sql\n-- Function to validate connection is properly set up\nCREATE OR REPLACE FUNCTION api.validate_connection()\nRETURNS jsonb\nLANGUAGE plpgsql\nSTABLE\nAS $\nDECLARE\n l_result jsonb;\nBEGIN\n l_result := jsonb_build_object(\n 'user_id', current_setting('myapp.current_user_id', true),\n 'tenant_id', current_setting('myapp.current_tenant_id', true),\n 'application_name', current_setting('application_name', true),\n 'server_version', current_setting('server_version'),\n 'timezone', current_setting('timezone'),\n 'search_path', current_setting('search_path'),\n 'is_superuser', current_setting('is_superuser')\n );\n \n RETURN l_result;\nEND;\n$;\n```\n\n### Pool Sizing\n\nThe most common mistake is making the pool too large. A small, well-tuned pool outperforms a large one because fewer connections mean less context switching, less lock contention, and better CPU cache utilization.\n\n**Formula**: `pool_size = (CPU cores * 2) + effective_spindle_count`\n\nFor SSDs or cloud storage (no spinning disks), simplify to: `pool_size = CPU cores * 2`\n\n| Server | CPU Cores | SSD? | Pool Size |\n|--------|-----------|------|-----------|\n| Small (cloud) | 4 | Yes | 8 |\n| Medium | 8 | Yes | 16 |\n| Large | 16 | Yes | 32 |\n| Large (HDD) | 16 | No (4 disks) | 36 |\n\n> These are **database connection** pool sizes (PgBouncer `default_pool_size`). The client-side pool (`max_client_conn`) can be much larger since PgBouncer multiplexes clients onto fewer server connections.\n\n### max_connections Tuning\n\nThe default `max_connections = 100` is often sufficient. Each connection consumes ~5-10 MB of RAM (for `work_mem`, temp buffers, connection state). Increasing it carelessly wastes memory and hurts performance.\n\n```sql\n-- Check current usage vs limit\nSELECT\n current_setting('max_connections')::int AS max_connections,\n (SELECT COUNT(*) FROM pg_stat_activity) AS current_connections,\n round(100.0 * (SELECT COUNT(*) FROM pg_stat_activity)\n / current_setting('max_connections')::int, 1) AS utilization_pct;\n\n-- Reserve connections for superuser access\n-- In postgresql.conf:\n-- superuser_reserved_connections = 3 (default)\n```\n\n**Rule of thumb**: With PgBouncer in front, keep `max_connections` low (2-4x your pool size) and let PgBouncer handle the client fan-out. There is no benefit to setting `max_connections = 1000` if only 32 connections are active at a time.\n\n### Idle Timeout Settings\n\nIdle connections waste resources. Use timeouts to reclaim them.\n\n```sql\n-- Kill transactions left open by accident (e.g., forgotten BEGIN without COMMIT)\nALTER SYSTEM SET idle_in_transaction_session_timeout = '60s';\n\n-- Kill completely idle connections after 10 minutes (PostgreSQL 14+)\n-- Useful for applications that open connections and forget to close them\nALTER SYSTEM SET idle_session_timeout = '10min';\n\nSELECT pg_reload_conf();\n```\n\n```ini\n# pgbouncer.ini\n\n# Reclaim server connections sitting idle in PgBouncer's pool\nserver_idle_timeout = 600 ; Close server connection after 10 min idle\n\n# Disconnect clients that have been idle too long (no active query or txn)\nclient_idle_timeout = 0 ; 0 = disabled (default); set to e.g. 3600 for 1 hour\n\n# Time to wait for a server connection before giving up\nserver_connect_timeout = 15\n```\n\n### Connection Limits Per Role\n\nPrevent a single application or role from monopolizing all connections.\n\n```sql\n-- Limit the app service role to 50 connections\nALTER ROLE app_service CONNECTION LIMIT 50;\n\n-- Limit a reporting role to fewer connections\nALTER ROLE app_reporting CONNECTION LIMIT 10;\n\n-- Check current limits\nSELECT rolname, rolconnlimit\nFROM pg_roles\nWHERE rolconnlimit > 0;\n```\n\n### Monitoring Pool Utilization\n\nUse `pg_stat_activity` to monitor connection usage (see also `monitoring-observability.md` §Connection Monitoring for a full `api.get_connection_pool_status()` function).\n\n```sql\n-- Connection summary by state and application\nSELECT\n application_name,\n state,\n COUNT(*) AS connections\nFROM pg_stat_activity\nWHERE datname = current_database()\nGROUP BY application_name, state\nORDER BY connections DESC;\n\n-- Find connections idle in transaction (potential pool leaks)\nSELECT\n pid,\n usename,\n application_name,\n state,\n now() - state_change AS idle_duration,\n left(query, 80) AS last_query\nFROM pg_stat_activity\nWHERE state = 'idle in transaction'\n AND state_change \u003c now() - interval '30 seconds'\nORDER BY state_change;\n```\n\n## Session Configuration\n\n### Role-Based Defaults\n\n```sql\n-- Set defaults for application role\nALTER ROLE app_service SET search_path = api, pg_temp;\nALTER ROLE app_service SET statement_timeout = '30s';\nALTER ROLE app_service SET lock_timeout = '10s';\nALTER ROLE app_service SET idle_in_transaction_session_timeout = '60s';\nALTER ROLE app_service SET timezone = 'UTC';\n\n-- Set custom defaults\nALTER ROLE app_service SET myapp.default_page_size = '25';\n```\n\n### Dynamic Configuration\n\n```sql\n-- Set session-level configuration\nSET statement_timeout = '10s';\nSET work_mem = '256MB'; -- For complex queries in this session\nSET enable_seqscan = off; -- For testing indexes\n\n-- Reset to default\nRESET statement_timeout;\nRESET ALL;\n\n-- Check current settings\nSHOW statement_timeout;\nSHOW ALL;\n```\n\n### Application-Specific Settings\n\n```sql\n-- Create settings table for application configuration\nCREATE TABLE data.app_settings (\n key text PRIMARY KEY,\n value text NOT NULL,\n description text,\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Load settings into session\nCREATE OR REPLACE FUNCTION api.load_app_settings()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_setting RECORD;\nBEGIN\n FOR l_setting IN SELECT key, value FROM data.app_settings LOOP\n PERFORM set_config('myapp.' || l_setting.key, l_setting.value, false);\n END LOOP;\nEND;\n$;\n\n-- Get setting with fallback\nCREATE OR REPLACE FUNCTION private.get_app_setting(\n in_key text,\n in_default text DEFAULT NULL\n)\nRETURNS text\nLANGUAGE sql\nSTABLE\nAS $\n SELECT COALESCE(\n NULLIF(current_setting('myapp.' || in_key, true), ''),\n in_default\n );\n$;\n```\n\n## Connection Lifecycle\n\n### Connection Initialization\n\n```sql\n-- Procedure to initialize new connections\nCREATE OR REPLACE FUNCTION api.init_connection()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Load application settings\n PERFORM api.load_app_settings();\n \n -- Set standard configuration\n SET timezone = 'UTC';\n SET datestyle = 'ISO, MDY';\n \n -- Log connection\n INSERT INTO data.connection_log (\n connected_at,\n client_addr,\n application_name,\n backend_pid\n ) VALUES (\n now(),\n inet_client_addr(),\n current_setting('application_name', true),\n pg_backend_pid()\n );\nEND;\n$;\n```\n\n### Idle Connection Management\n\n```sql\n-- Find and terminate idle connections\nCREATE OR REPLACE FUNCTION api.cleanup_idle_connections(\n in_idle_timeout interval DEFAULT interval '10 minutes'\n)\nRETURNS integer\nLANGUAGE plpgsql\nSECURITY DEFINER\nAS $\nDECLARE\n l_terminated integer := 0;\n l_pid integer;\nBEGIN\n FOR l_pid IN\n SELECT pid\n FROM pg_stat_activity\n WHERE state = 'idle'\n AND state_change \u003c now() - in_idle_timeout\n AND pid != pg_backend_pid()\n AND usename = 'app_service'\n LOOP\n PERFORM pg_terminate_backend(l_pid);\n l_terminated := l_terminated + 1;\n END LOOP;\n \n RETURN l_terminated;\nEND;\n$;\n```\n\n### Connection Statistics\n\n```sql\n-- View current connections\nCREATE OR REPLACE VIEW api.v_connection_stats AS\nSELECT \n usename AS username,\n application_name,\n client_addr,\n state,\n state_change,\n now() - state_change AS idle_time,\n query,\n wait_event_type,\n wait_event\nFROM pg_stat_activity\nWHERE datname = current_database()\nORDER BY state_change;\n\n-- Connection pool status\nCREATE OR REPLACE FUNCTION api.get_connection_summary()\nRETURNS TABLE (\n state text,\n count bigint,\n avg_idle_seconds numeric\n)\nLANGUAGE sql\nSTABLE\nAS $\n SELECT \n state,\n COUNT(*),\n ROUND(AVG(EXTRACT(EPOCH FROM (now() - state_change)))::numeric, 2)\n FROM pg_stat_activity\n WHERE datname = current_database()\n GROUP BY state;\n$;\n```\n\n## Security Context\n\n### Multi-Tenant Security\n\n```sql\n-- Ensure tenant isolation in all queries\nCREATE OR REPLACE FUNCTION private.require_tenant_context()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_tenant_id uuid;\nBEGIN\n l_tenant_id := private.get_current_tenant_id();\n \n IF l_tenant_id IS NULL THEN\n RAISE EXCEPTION 'Tenant context not set'\n USING ERRCODE = 'P0051';\n END IF;\nEND;\n$;\n\n-- Verify user belongs to tenant\nCREATE OR REPLACE FUNCTION private.verify_user_tenant_access()\nRETURNS boolean\nLANGUAGE plpgsql\nSTABLE\nAS $\nDECLARE\n l_user_id uuid;\n l_tenant_id uuid;\n l_has_access boolean;\nBEGIN\n l_user_id := private.get_current_user_id();\n l_tenant_id := private.get_current_tenant_id();\n \n SELECT EXISTS(\n SELECT 1 FROM data.user_tenants\n WHERE user_id = l_user_id\n AND tenant_id = l_tenant_id\n AND is_active = true\n ) INTO l_has_access;\n \n IF NOT l_has_access THEN\n RAISE EXCEPTION 'User does not have access to tenant'\n USING ERRCODE = 'P0052';\n END IF;\n \n RETURN true;\nEND;\n$;\n```\n\n### Impersonation\n\n```sql\n-- Allow admins to impersonate other users\nCREATE OR REPLACE FUNCTION api.impersonate_user(\n in_target_user_id uuid\n)\nRETURNS void\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_current_user_id uuid;\n l_is_admin boolean;\nBEGIN\n l_current_user_id := private.get_current_user_id();\n \n -- Check if current user is admin\n SELECT role = 'admin' INTO l_is_admin\n FROM data.users\n WHERE id = l_current_user_id;\n \n IF NOT l_is_admin THEN\n RAISE EXCEPTION 'Only admins can impersonate users'\n USING ERRCODE = 'P0053';\n END IF;\n \n -- Store original user for audit\n PERFORM set_config('myapp.original_user_id', l_current_user_id::text, false);\n \n -- Set impersonated user\n PERFORM set_config('myapp.current_user_id', in_target_user_id::text, false);\n PERFORM set_config('myapp.is_impersonating', 'true', false);\n \n -- Log impersonation\n INSERT INTO app_audit.impersonation_log (\n admin_user_id, target_user_id, started_at\n ) VALUES (\n l_current_user_id, in_target_user_id, now()\n );\nEND;\n$;\n\n-- End impersonation\nCREATE OR REPLACE FUNCTION api.end_impersonation()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_original_user_id text;\nBEGIN\n l_original_user_id := current_setting('myapp.original_user_id', true);\n \n IF l_original_user_id IS NULL OR l_original_user_id = '' THEN\n RAISE EXCEPTION 'Not currently impersonating';\n END IF;\n \n -- Restore original user\n PERFORM set_config('myapp.current_user_id', l_original_user_id, false);\n PERFORM set_config('myapp.original_user_id', '', false);\n PERFORM set_config('myapp.is_impersonating', 'false', false);\nEND;\n$;\n```\n\n### Rate Limiting\n\n```sql\n-- Simple rate limiting using session tracking\nCREATE TABLE data.rate_limits (\n user_id uuid NOT NULL,\n action text NOT NULL,\n window_start timestamptz NOT NULL,\n count integer NOT NULL DEFAULT 1,\n PRIMARY KEY (user_id, action, window_start)\n);\n\nCREATE OR REPLACE FUNCTION private.check_rate_limit(\n in_action text,\n in_max_requests integer,\n in_window_seconds integer\n)\nRETURNS boolean\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_user_id uuid;\n l_window_start timestamptz;\n l_current_count integer;\nBEGIN\n l_user_id := private.get_current_user_id();\n l_window_start := date_trunc('second', now()) \n - ((EXTRACT(EPOCH FROM now())::integer % in_window_seconds) * interval '1 second');\n \n -- Upsert rate limit counter\n INSERT INTO data.rate_limits (user_id, action, window_start, count)\n VALUES (l_user_id, in_action, l_window_start, 1)\n ON CONFLICT (user_id, action, window_start) DO UPDATE\n SET count = data.rate_limits.count + 1\n RETURNING count INTO l_current_count;\n \n -- Check limit\n IF l_current_count > in_max_requests THEN\n RAISE EXCEPTION 'Rate limit exceeded for action: %', in_action\n USING ERRCODE = 'P0054';\n END IF;\n \n RETURN true;\nEND;\n$;\n\n-- Usage in API functions\nCREATE OR REPLACE FUNCTION api.send_email(in_to text, in_subject text, in_body text)\nRETURNS void\nLANGUAGE plpgsql\nSECURITY DEFINER\nAS $\nBEGIN\n -- Check rate limit: 10 emails per minute\n PERFORM private.check_rate_limit('send_email', 10, 60);\n \n -- Send email logic...\nEND;\n$;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":19278,"content_sha256":"a3d76ce5094259b2db0504e0c23b070befbae91051212503210f5f835c6d7293"},{"filename":"references/testing-patterns.md","content":"# Testing Patterns for PostgreSQL\n\nThis document covers unit testing, integration testing, and test data management for PostgreSQL using pgTAP and native patterns.\n\n## Table of Contents\n\n1. [pgTAP Setup](#pgtap-setup)\n2. [Test Structure](#test-structure)\n3. [Testing Functions](#testing-functions)\n4. [Testing Procedures](#testing-procedures)\n5. [Testing Triggers](#testing-triggers)\n6. [Testing Constraints](#testing-constraints)\n7. [Test Data Management](#test-data-management)\n8. [Transaction Isolation](#transaction-isolation)\n9. [Migration Testing](#migration-testing)\n10. [CI/CD Integration](#cicd-integration)\n\n## pgTAP Setup\n\n### Installation\n\n```sql\n-- Install pgTAP extension\nCREATE EXTENSION IF NOT EXISTS pgtap;\n\n-- Create test schema\nCREATE SCHEMA IF NOT EXISTS test;\nCOMMENT ON SCHEMA test IS 'Unit tests using pgTAP';\n```\n\n### Test Runner Schema\n\n```sql\n-- Track test execution\nCREATE TABLE test.test_runs (\n id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n run_at timestamptz NOT NULL DEFAULT now(),\n total_tests integer NOT NULL,\n passed integer NOT NULL,\n failed integer NOT NULL,\n execution_ms integer,\n details jsonb\n);\n```\n\n## Test Structure\n\n### Basic Test Template\n\n```sql\n-- ============================================================================\n-- Test: test.test_api_get_customer\n-- Tests: api.get_customer function\n-- ============================================================================\nCREATE OR REPLACE FUNCTION test.test_api_get_customer()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_customer_id uuid;\n l_result RECORD;\nBEGIN\n -- ========================================\n -- ARRANGE: Set up test data\n -- ========================================\n INSERT INTO data.customers (email, name, password_hash)\n VALUES ('[email protected]', 'Test User', 'hash123')\n RETURNING id INTO l_customer_id;\n \n -- ========================================\n -- ACT: Call the function under test\n -- ========================================\n SELECT * INTO l_result\n FROM api.get_customer(l_customer_id);\n \n -- ========================================\n -- ASSERT: Verify results\n -- ========================================\n RETURN NEXT ok(l_result.id IS NOT NULL, 'Should return customer ID');\n RETURN NEXT is(l_result.email, '[email protected]', 'Should return correct email');\n RETURN NEXT is(l_result.name, 'Test User', 'Should return correct name');\n \n -- Test non-existent customer\n SELECT * INTO l_result\n FROM api.get_customer('00000000-0000-0000-0000-000000000000'::uuid);\n \n RETURN NEXT ok(l_result.id IS NULL, 'Should return NULL for non-existent customer');\n \nEND;\n$;\n```\n\n### Test Naming Convention\n\n```sql\n-- Pattern: test.test_{schema}_{function_name}[_{scenario}]\n\ntest.test_api_get_customer()\ntest.test_api_get_customer_not_found()\ntest.test_api_insert_customer()\ntest.test_api_insert_customer_duplicate_email()\ntest.test_private_hash_password()\ntest.test_trigger_set_updated_at()\n```\n\n## Testing Functions\n\n### Testing Read Functions\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_api_select_orders_by_customer()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_customer_id uuid;\n l_order_ids uuid[];\n l_results RECORD;\n l_count integer;\nBEGIN\n -- ARRANGE\n INSERT INTO data.customers (email, name, password_hash)\n VALUES ('[email protected]', 'Test Customer', 'hash')\n RETURNING id INTO l_customer_id;\n \n -- Create multiple orders\n INSERT INTO data.orders (customer_id, status, total)\n VALUES \n (l_customer_id, 'pending', 100.00),\n (l_customer_id, 'shipped', 200.00),\n (l_customer_id, 'delivered', 150.00)\n RETURNING ARRAY_AGG(id) INTO l_order_ids;\n \n -- ACT & ASSERT: Test without filter\n SELECT COUNT(*) INTO l_count\n FROM api.select_orders_by_customer(l_customer_id);\n \n RETURN NEXT is(l_count, 3, 'Should return all 3 orders');\n \n -- ACT & ASSERT: Test with status filter\n SELECT COUNT(*) INTO l_count\n FROM api.select_orders_by_customer(l_customer_id, 'pending');\n \n RETURN NEXT is(l_count, 1, 'Should return only pending orders');\n \n -- ACT & ASSERT: Test with limit\n SELECT COUNT(*) INTO l_count\n FROM api.select_orders_by_customer(l_customer_id, in_limit := 2);\n \n RETURN NEXT is(l_count, 2, 'Should respect limit parameter');\n \n -- ACT & ASSERT: Test ordering (most recent first)\n FOR l_results IN \n SELECT created_at FROM api.select_orders_by_customer(l_customer_id)\n LOOP\n -- Just verify we get results; ordering tested implicitly\n RETURN NEXT ok(l_results.created_at IS NOT NULL, 'Should have created_at');\n END LOOP;\n \nEND;\n$;\n```\n\n### Testing Functions That Return Computed Values\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_private_ord_calculate_total()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_customer_id uuid;\n l_order_id uuid;\n l_total numeric;\nBEGIN\n -- ARRANGE\n INSERT INTO data.customers (email, name, password_hash)\n VALUES ('[email protected]', 'Calc Test', 'hash')\n RETURNING id INTO l_customer_id;\n \n INSERT INTO data.orders (customer_id, status, total)\n VALUES (l_customer_id, 'pending', 0)\n RETURNING id INTO l_order_id;\n \n INSERT INTO data.order_items (order_id, product_name, quantity, unit_price)\n VALUES \n (l_order_id, 'Widget A', 2, 10.00), -- 20.00\n (l_order_id, 'Widget B', 3, 15.50), -- 46.50\n (l_order_id, 'Widget C', 1, 100.00); -- 100.00\n -- Total: 166.50\n \n -- ACT\n l_total := private.ord_calculate_total(l_order_id);\n \n -- ASSERT\n RETURN NEXT is(l_total, 166.50::numeric, 'Should calculate correct total');\n \n -- Test empty order\n INSERT INTO data.orders (customer_id, status, total)\n VALUES (l_customer_id, 'pending', 0)\n RETURNING id INTO l_order_id;\n \n l_total := private.ord_calculate_total(l_order_id);\n \n RETURN NEXT is(l_total, 0::numeric, 'Empty order should have zero total');\n \nEND;\n$;\n```\n\n## Testing Procedures\n\n### Testing Insert Procedures\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_api_insert_customer()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_id uuid;\n l_result RECORD;\nBEGIN\n -- ACT\n CALL api.insert_customer(\n in_email := '[email protected]',\n in_name := 'New Customer',\n in_password := 'securepass123',\n io_id := l_id\n );\n \n -- ASSERT: ID was returned\n RETURN NEXT ok(l_id IS NOT NULL, 'Should return generated ID');\n \n -- ASSERT: Data was inserted correctly\n SELECT * INTO l_result FROM data.customers WHERE id = l_id;\n \n RETURN NEXT is(l_result.email, '[email protected]', 'Email should be stored lowercase');\n RETURN NEXT is(l_result.name, 'New Customer', 'Name should be stored');\n RETURN NEXT ok(l_result.password_hash IS NOT NULL, 'Password should be hashed');\n RETURN NEXT isnt(l_result.password_hash, 'securepass123', 'Password should not be plaintext');\n RETURN NEXT ok(l_result.is_active, 'Should default to active');\n RETURN NEXT ok(l_result.created_at IS NOT NULL, 'Should have created_at');\n \nEND;\n$;\n```\n\n### Testing Procedures That Should Fail\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_api_insert_customer_duplicate_email()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_id uuid;\n l_exception_thrown boolean := false;\n l_sqlstate text;\nBEGIN\n -- ARRANGE: Create first customer\n CALL api.insert_customer(\n in_email := '[email protected]',\n in_name := 'First Customer',\n in_password := 'pass123',\n io_id := l_id\n );\n \n -- ACT & ASSERT: Try to create duplicate\n BEGIN\n CALL api.insert_customer(\n in_email := '[email protected]', -- Same email\n in_name := 'Second Customer',\n in_password := 'pass456',\n io_id := l_id\n );\n EXCEPTION\n WHEN unique_violation THEN\n l_exception_thrown := true;\n l_sqlstate := SQLSTATE;\n WHEN OTHERS THEN\n l_exception_thrown := true;\n l_sqlstate := SQLSTATE;\n END;\n \n RETURN NEXT ok(l_exception_thrown, 'Should throw exception for duplicate email');\n RETURN NEXT is(l_sqlstate, '23505', 'Should be unique_violation error');\n \nEND;\n$;\n```\n\n### Testing Update Procedures\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_api_ord_update_status()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_customer_id uuid;\n l_order_id uuid;\n l_status text;\n l_exception_thrown boolean;\nBEGIN\n -- ARRANGE\n INSERT INTO data.customers (email, name, password_hash)\n VALUES ('[email protected]', 'Status Test', 'hash')\n RETURNING id INTO l_customer_id;\n \n INSERT INTO data.orders (customer_id, status, total)\n VALUES (l_customer_id, 'pending', 100.00)\n RETURNING id INTO l_order_id;\n \n -- ACT: Valid transition pending -> confirmed\n CALL api.ord_update_status(l_order_id, 'confirmed');\n \n SELECT status INTO l_status FROM data.orders WHERE id = l_order_id;\n RETURN NEXT is(l_status, 'confirmed', 'Should update to confirmed');\n \n -- ACT: Valid transition confirmed -> processing\n CALL api.ord_update_status(l_order_id, 'processing');\n \n SELECT status INTO l_status FROM data.orders WHERE id = l_order_id;\n RETURN NEXT is(l_status, 'processing', 'Should update to processing');\n \n -- ACT & ASSERT: Invalid transition processing -> pending\n l_exception_thrown := false;\n BEGIN\n CALL api.ord_update_status(l_order_id, 'pending');\n EXCEPTION\n WHEN OTHERS THEN\n l_exception_thrown := true;\n END;\n \n RETURN NEXT ok(l_exception_thrown, 'Should reject invalid status transition');\n \n -- Verify status unchanged\n SELECT status INTO l_status FROM data.orders WHERE id = l_order_id;\n RETURN NEXT is(l_status, 'processing', 'Status should remain unchanged after failed transition');\n \nEND;\n$;\n```\n\n## Testing Triggers\n\n### Testing updated_at Trigger\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_trigger_set_updated_at()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_customer_id uuid;\n l_created_at timestamptz;\n l_updated_at_before timestamptz;\n l_updated_at_after timestamptz;\nBEGIN\n -- ARRANGE\n INSERT INTO data.customers (email, name, password_hash)\n VALUES ('[email protected]', 'Trigger Test', 'hash')\n RETURNING id, created_at, updated_at \n INTO l_customer_id, l_created_at, l_updated_at_before;\n \n -- Verify initial state\n RETURN NEXT is(l_created_at, l_updated_at_before, \n 'created_at and updated_at should match initially');\n \n -- Wait a tiny bit to ensure timestamp difference\n PERFORM pg_sleep(0.01);\n \n -- ACT\n UPDATE data.customers SET name = 'Updated Name' WHERE id = l_customer_id;\n \n SELECT updated_at INTO l_updated_at_after \n FROM data.customers WHERE id = l_customer_id;\n \n -- ASSERT\n RETURN NEXT ok(l_updated_at_after > l_updated_at_before, \n 'updated_at should be updated by trigger');\n \n RETURN NEXT is(\n (SELECT created_at FROM data.customers WHERE id = l_customer_id),\n l_created_at,\n 'created_at should not change'\n );\n \nEND;\n$;\n```\n\n### Testing Audit Triggers\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_trigger_audit_log()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_customer_id uuid;\n l_audit_count integer;\n l_audit_record RECORD;\nBEGIN\n -- ARRANGE: Clear audit log for this test\n DELETE FROM app_audit.changelog WHERE table_name = 'customers';\n \n -- ACT: Insert\n INSERT INTO data.customers (email, name, password_hash)\n VALUES ('[email protected]', 'Audit Test', 'hash')\n RETURNING id INTO l_customer_id;\n \n -- ASSERT: Insert logged\n SELECT COUNT(*) INTO l_audit_count \n FROM app_audit.changelog \n WHERE table_name = 'customers' AND operation = 'INSERT';\n \n RETURN NEXT is(l_audit_count, 1, 'Insert should be logged');\n \n -- ACT: Update\n UPDATE data.customers SET name = 'Updated Name' WHERE id = l_customer_id;\n \n -- ASSERT: Update logged\n SELECT * INTO l_audit_record \n FROM app_audit.changelog \n WHERE table_name = 'customers' AND operation = 'UPDATE'\n ORDER BY changed_at DESC LIMIT 1;\n \n RETURN NEXT ok(l_audit_record.old_values IS NOT NULL, 'Should log old values');\n RETURN NEXT ok(l_audit_record.new_values IS NOT NULL, 'Should log new values');\n \n -- ACT: Delete\n DELETE FROM data.customers WHERE id = l_customer_id;\n \n -- ASSERT: Delete logged\n SELECT COUNT(*) INTO l_audit_count \n FROM app_audit.changelog \n WHERE table_name = 'customers' AND operation = 'DELETE';\n \n RETURN NEXT is(l_audit_count, 1, 'Delete should be logged');\n \nEND;\n$;\n```\n\n## Testing Constraints\n\n### Testing Check Constraints\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_constraint_order_total_positive()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_customer_id uuid;\n l_exception_thrown boolean := false;\nBEGIN\n -- ARRANGE\n INSERT INTO data.customers (email, name, password_hash)\n VALUES ('[email protected]', 'Constraint Test', 'hash')\n RETURNING id INTO l_customer_id;\n \n -- ACT & ASSERT: Positive total should work\n BEGIN\n INSERT INTO data.orders (customer_id, status, total)\n VALUES (l_customer_id, 'pending', 100.00);\n \n RETURN NEXT pass('Positive total should be accepted');\n EXCEPTION\n WHEN OTHERS THEN\n RETURN NEXT fail('Positive total should be accepted');\n END;\n \n -- ACT & ASSERT: Zero total should work\n BEGIN\n INSERT INTO data.orders (customer_id, status, total)\n VALUES (l_customer_id, 'pending', 0.00);\n \n RETURN NEXT pass('Zero total should be accepted');\n EXCEPTION\n WHEN OTHERS THEN\n RETURN NEXT fail('Zero total should be accepted');\n END;\n \n -- ACT & ASSERT: Negative total should fail\n BEGIN\n INSERT INTO data.orders (customer_id, status, total)\n VALUES (l_customer_id, 'pending', -50.00);\n \n RETURN NEXT fail('Negative total should be rejected');\n EXCEPTION\n WHEN check_violation THEN\n RETURN NEXT pass('Negative total correctly rejected');\n WHEN OTHERS THEN\n RETURN NEXT fail('Wrong exception type for negative total');\n END;\n \nEND;\n$;\n```\n\n### Testing Foreign Key Constraints\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_constraint_order_customer_fk()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_exception_thrown boolean := false;\n l_fake_customer_id uuid := gen_random_uuid();\nBEGIN\n -- ACT & ASSERT: Non-existent customer should fail\n BEGIN\n INSERT INTO data.orders (customer_id, status, total)\n VALUES (l_fake_customer_id, 'pending', 100.00);\n \n RETURN NEXT fail('Should reject non-existent customer');\n EXCEPTION\n WHEN foreign_key_violation THEN\n RETURN NEXT pass('Correctly rejected non-existent customer');\n WHEN OTHERS THEN\n RETURN NEXT fail('Wrong exception type: ' || SQLERRM);\n END;\n \nEND;\n$;\n```\n\n## Test Data Management\n\n### Test Data Factory\n\n```sql\n-- ============================================================================\n-- Test Data Factory Functions\n-- ============================================================================\n\nCREATE OR REPLACE FUNCTION test.create_customer(\n in_email text DEFAULT NULL,\n in_name text DEFAULT 'Test Customer',\n in_is_active boolean DEFAULT true\n)\nRETURNS uuid\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_id uuid;\n l_email text;\nBEGIN\n l_email := COALESCE(in_email, 'test_' || gen_random_uuid()::text || '@example.com');\n \n INSERT INTO data.customers (email, name, password_hash, is_active)\n VALUES (l_email, in_name, 'test_hash', in_is_active)\n RETURNING id INTO l_id;\n \n RETURN l_id;\nEND;\n$;\n\nCREATE OR REPLACE FUNCTION test.create_order(\n in_customer_id uuid DEFAULT NULL,\n in_status text DEFAULT 'pending',\n in_total numeric DEFAULT 100.00\n)\nRETURNS uuid\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_id uuid;\n l_customer_id uuid;\nBEGIN\n -- Create customer if not provided\n l_customer_id := COALESCE(in_customer_id, test.create_customer());\n \n INSERT INTO data.orders (customer_id, status, total)\n VALUES (l_customer_id, in_status, in_total)\n RETURNING id INTO l_id;\n \n RETURN l_id;\nEND;\n$;\n\nCREATE OR REPLACE FUNCTION test.create_order_with_items(\n in_customer_id uuid DEFAULT NULL,\n in_item_count integer DEFAULT 3\n)\nRETURNS uuid\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_order_id uuid;\n l_customer_id uuid;\n i integer;\nBEGIN\n l_customer_id := COALESCE(in_customer_id, test.create_customer());\n \n INSERT INTO data.orders (customer_id, status, total)\n VALUES (l_customer_id, 'pending', 0)\n RETURNING id INTO l_order_id;\n \n FOR i IN 1..in_item_count LOOP\n INSERT INTO data.order_items (order_id, product_name, quantity, unit_price)\n VALUES (l_order_id, 'Product ' || i, i, i * 10.00);\n END LOOP;\n \n -- Update total\n UPDATE data.orders \n SET total = (SELECT COALESCE(SUM(quantity * unit_price), 0) \n FROM data.order_items WHERE order_id = l_order_id)\n WHERE id = l_order_id;\n \n RETURN l_order_id;\nEND;\n$;\n```\n\n### Cleanup Functions\n\n```sql\nCREATE OR REPLACE FUNCTION test.cleanup_test_data()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Delete in correct order (respect FKs)\n DELETE FROM data.order_items WHERE order_id IN (\n SELECT id FROM data.orders WHERE customer_id IN (\n SELECT id FROM data.customers WHERE email LIKE 'test_%@example.com'\n )\n );\n DELETE FROM data.orders WHERE customer_id IN (\n SELECT id FROM data.customers WHERE email LIKE 'test_%@example.com'\n );\n DELETE FROM data.customers WHERE email LIKE 'test_%@example.com';\n DELETE FROM data.customers WHERE email LIKE '%@test.com';\nEND;\n$;\n```\n\n## Transaction Isolation\n\n### Running Tests in Transactions (Rollback Pattern)\n\n```sql\nCREATE OR REPLACE FUNCTION test.run_test_isolated(in_test_function text)\nRETURNS TABLE (test_result text)\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Start savepoint\n -- Execute test\n -- Rollback to savepoint\n \n RETURN QUERY EXECUTE format(\n 'SELECT * FROM %s()',\n in_test_function\n );\n \n -- Note: In practice, wrap in transaction from caller\nEND;\n$;\n\n-- Usage from psql:\n-- BEGIN;\n-- SELECT * FROM test.test_api_insert_customer();\n-- ROLLBACK;\n```\n\n### Test Runner with Automatic Rollback\n\n```sql\nCREATE OR REPLACE FUNCTION test.run_all_tests()\nRETURNS TABLE (\n test_name text,\n result text,\n message text\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_test RECORD;\n l_result RECORD;\n l_start_time timestamptz;\n l_total integer := 0;\n l_passed integer := 0;\n l_failed integer := 0;\nBEGIN\n l_start_time := clock_timestamp();\n \n -- Find all test functions\n FOR l_test IN\n SELECT p.proname AS name\n FROM pg_proc p\n JOIN pg_namespace n ON p.pronamespace = n.oid\n WHERE n.nspname = 'test'\n AND p.proname LIKE 'test_%'\n ORDER BY p.proname\n LOOP\n -- Run each test in a subtransaction\n BEGIN\n FOR l_result IN EXECUTE format('SELECT * FROM test.%I()', l_test.name)\n LOOP\n l_total := l_total + 1;\n \n IF l_result::text LIKE 'ok%' OR l_result::text LIKE 'pass%' THEN\n l_passed := l_passed + 1;\n RETURN QUERY SELECT l_test.name, 'PASS'::text, l_result::text;\n ELSE\n l_failed := l_failed + 1;\n RETURN QUERY SELECT l_test.name, 'FAIL'::text, l_result::text;\n END IF;\n END LOOP;\n EXCEPTION\n WHEN OTHERS THEN\n l_failed := l_failed + 1;\n RETURN QUERY SELECT l_test.name, 'ERROR'::text, SQLERRM;\n END;\n END LOOP;\n \n -- Log results\n INSERT INTO test.test_runs (total_tests, passed, failed, execution_ms)\n VALUES (\n l_total, \n l_passed, \n l_failed,\n EXTRACT(MILLISECONDS FROM clock_timestamp() - l_start_time)::integer\n );\n \n -- Summary row\n RETURN QUERY SELECT \n '=== SUMMARY ==='::text,\n CASE WHEN l_failed = 0 THEN 'ALL PASSED' ELSE 'FAILURES' END,\n format('%s total, %s passed, %s failed', l_total, l_passed, l_failed);\n \nEND;\n$;\n```\n\n## Migration Testing\n\n### Testing Migration Applies Correctly\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_migration_001_creates_customers()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Verify table exists\n RETURN NEXT ok(\n EXISTS(\n SELECT 1 FROM information_schema.tables \n WHERE table_schema = 'data' AND table_name = 'customers'\n ),\n 'customers table should exist'\n );\n \n -- Verify columns\n RETURN NEXT ok(\n EXISTS(\n SELECT 1 FROM information_schema.columns \n WHERE table_schema = 'data' \n AND table_name = 'customers' \n AND column_name = 'id'\n AND data_type = 'uuid'\n ),\n 'customers.id should be uuid'\n );\n \n -- Verify constraints\n RETURN NEXT ok(\n EXISTS(\n SELECT 1 FROM information_schema.table_constraints\n WHERE table_schema = 'data'\n AND table_name = 'customers'\n AND constraint_type = 'PRIMARY KEY'\n ),\n 'customers should have primary key'\n );\n \n -- Verify indexes\n RETURN NEXT ok(\n EXISTS(\n SELECT 1 FROM pg_indexes\n WHERE schemaname = 'data'\n AND tablename = 'customers'\n AND indexname = 'customers_email_key'\n ),\n 'customers should have email unique index'\n );\n \nEND;\n$;\n```\n\n### Testing Rollback\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_migration_rollback()\nRETURNS SETOF text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_version_before integer;\n l_version_after integer;\nBEGIN\n -- Get current version\n SELECT COUNT(*) INTO l_version_before \n FROM app_migration.changelog \n WHERE type = 'versioned' AND success = true;\n \n -- Apply a test migration\n SELECT app_migration.acquire_lock();\n \n CALL app_migration.run_versioned(\n in_version := '999',\n in_description := 'Test migration for rollback',\n in_sql := 'CREATE TABLE data.test_rollback_table (id int);',\n in_rollback_sql := 'DROP TABLE IF EXISTS data.test_rollback_table;'\n );\n \n -- Verify table created\n RETURN NEXT ok(\n EXISTS(SELECT 1 FROM information_schema.tables \n WHERE table_schema = 'data' AND table_name = 'test_rollback_table'),\n 'Test table should be created'\n );\n \n -- Rollback\n CALL app_migration.rollback('999');\n \n -- Verify table removed\n RETURN NEXT ok(\n NOT EXISTS(SELECT 1 FROM information_schema.tables \n WHERE table_schema = 'data' AND table_name = 'test_rollback_table'),\n 'Test table should be removed after rollback'\n );\n \n SELECT app_migration.release_lock();\n \nEND;\n$;\n```\n\n## CI/CD Integration\n\n### GitHub Actions Workflow\n\n```yaml\n# .github/workflows/db-tests.yml\nname: Database Tests\n\non:\n push:\n paths:\n - 'db/**'\n pull_request:\n paths:\n - 'db/**'\n\njobs:\n test:\n runs-on: ubuntu-latest\n \n services:\n postgres:\n image: postgres:18\n env:\n POSTGRES_USER: test\n POSTGRES_PASSWORD: test\n POSTGRES_DB: test_db\n ports:\n - 5432:5432\n options: >-\n --health-cmd pg_isready\n --health-interval 10s\n --health-timeout 5s\n --health-retries 5\n\n steps:\n - uses: actions/checkout@v4\n \n - name: Install pgTAP\n run: |\n sudo apt-get update\n sudo apt-get install -y postgresql-client pgtap\n \n - name: Run migrations\n env:\n PGHOST: localhost\n PGUSER: test\n PGPASSWORD: test\n PGDATABASE: test_db\n run: |\n psql -f db/scripts/001_install_migration_system.sql\n psql -f db/scripts/002_migration_runner_helpers.sql\n psql -f db/migrations/run_all.sql\n \n - name: Run tests\n env:\n PGHOST: localhost\n PGUSER: test\n PGPASSWORD: test\n PGDATABASE: test_db\n run: |\n psql -f db/tests/install_tests.sql\n psql -c \"SELECT * FROM test.run_all_tests();\" | tee test_results.txt\n \n - name: Check for failures\n run: |\n if grep -q \"FAIL\\|ERROR\" test_results.txt; then\n echo \"Tests failed!\"\n exit 1\n fi\n```\n\n### Docker Test Setup\n\n```dockerfile\n# Dockerfile.test\nFROM postgres:18\n\n# Install pgTAP\nRUN apt-get update && apt-get install -y \\\n postgresql-18-pgtap \\\n && rm -rf /var/lib/apt/lists/*\n\n# Copy initialization scripts\nCOPY db/scripts/*.sql /docker-entrypoint-initdb.d/01-scripts/\nCOPY db/migrations/*.sql /docker-entrypoint-initdb.d/02-migrations/\nCOPY db/tests/*.sql /docker-entrypoint-initdb.d/03-tests/\n\n# Copy test runner\nCOPY db/tests/run_tests.sh /docker-entrypoint-initdb.d/99-run-tests.sh\n```\n\n```bash\n#!/bin/bash\n# db/tests/run_tests.sh\n\nset -e\n\npsql -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\" \u003c\u003cEOF\nSELECT * FROM test.run_all_tests();\nEOF\n\n# Check exit status\nFAILED=$(psql -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\" -t -c \\\n \"SELECT failed FROM test.test_runs ORDER BY id DESC LIMIT 1;\")\n\nif [ \"$FAILED\" -gt 0 ]; then\n echo \"Tests failed!\"\n exit 1\nfi\n\necho \"All tests passed!\"\n```\n\n### Running Tests Locally\n\n```bash\n#!/bin/bash\n# scripts/run-db-tests.sh\n\n# Start test database\ndocker-compose -f docker-compose.test.yml up -d postgres\n\n# Wait for database\nuntil docker-compose -f docker-compose.test.yml exec -T postgres pg_isready; do\n sleep 1\ndone\n\n# Run migrations\ndocker-compose -f docker-compose.test.yml exec -T postgres \\\n psql -U test -d test_db -f /app/db/migrations/run_all.sql\n\n# Run tests\ndocker-compose -f docker-compose.test.yml exec -T postgres \\\n psql -U test -d test_db -c \"SELECT * FROM test.run_all_tests();\"\n\n# Cleanup\ndocker-compose -f docker-compose.test.yml down -v\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":26606,"content_sha256":"e9f212ca2b650f21b3aaac8b2556e0243a601c319f5f1e1717c4235ff4e1884b"},{"filename":"references/time-series.md","content":"# Time-Series Data Patterns\n\nThis document covers time-series data optimization in native PostgreSQL, including table design, partitioning, indexing strategies, and efficient querying patterns.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Table Design](#table-design)\n3. [Partitioning for Time-Series](#partitioning-for-time-series)\n4. [Indexing Strategies](#indexing-strategies)\n5. [Query Patterns](#query-patterns)\n6. [Downsampling & Aggregation](#downsampling--aggregation)\n7. [Retention Management](#retention-management)\n8. [Performance Optimization](#performance-optimization)\n\n## Overview\n\n### Time-Series Characteristics\n\n| Characteristic | Description | Impact |\n|---------------|-------------|--------|\n| Append-heavy | Mostly INSERTs, rare UPDATEs | Optimize for writes |\n| Time-ordered | Data arrives in time order | Use BRIN indexes |\n| Range queries | Query by time windows | Partition by time |\n| High cardinality | Many unique time points | Consider aggregation |\n| Retention | Old data less valuable | Drop old partitions |\n\n### When to Use Native PostgreSQL vs TimescaleDB\n\n| Scenario | Native PostgreSQL | TimescaleDB |\n|----------|-------------------|-------------|\n| \u003c 100M rows | ✅ Sufficient | Good |\n| Simple queries | ✅ Sufficient | Good |\n| Automatic partitioning | Manual | ✅ Automatic |\n| Compression | ❌ Limited | ✅ Excellent |\n| Continuous aggregates | Manual | ✅ Built-in |\n| Already using PG | ✅ No extra setup | Requires extension |\n\n## Table Design\n\n### Basic Time-Series Table\n\n```sql\nCREATE TABLE data.metrics (\n id uuid DEFAULT uuidv7(),\n device_id uuid NOT NULL,\n metric_name text NOT NULL,\n value double precision NOT NULL,\n recorded_at timestamptz NOT NULL DEFAULT now(),\n\n -- Composite primary key including time for partitioning\n PRIMARY KEY (device_id, recorded_at, id)\n);\n\n-- Comment on design decisions\nCOMMENT ON TABLE data.metrics IS 'Time-series metrics data, partitioned by month';\n```\n\n### Wide Table Design (Multiple Metrics)\n\n```sql\n-- Wide table: one row per timestamp with multiple values\nCREATE TABLE data.sensor_readings (\n id uuid DEFAULT uuidv7(),\n sensor_id uuid NOT NULL,\n recorded_at timestamptz NOT NULL DEFAULT now(),\n\n -- Multiple metrics per row\n temperature double precision,\n humidity double precision,\n pressure double precision,\n battery_level smallint,\n\n PRIMARY KEY (sensor_id, recorded_at)\n);\n```\n\n### Narrow Table Design (EAV-Style)\n\n```sql\n-- Narrow table: one row per metric per timestamp\nCREATE TABLE data.measurements (\n id uuid DEFAULT uuidv7(),\n entity_id uuid NOT NULL,\n metric_name text NOT NULL,\n recorded_at timestamptz NOT NULL DEFAULT now(),\n value double precision NOT NULL,\n\n PRIMARY KEY (entity_id, metric_name, recorded_at)\n);\n\n-- Use when metrics are dynamic or sparse\n```\n\n### Optimized Columnar Layout\n\n```sql\n-- Ordered columns for better compression\nCREATE TABLE data.events (\n -- High cardinality, frequently filtered\n recorded_at timestamptz NOT NULL,\n event_type text NOT NULL,\n\n -- Foreign key\n device_id uuid NOT NULL,\n\n -- Payload (larger, variable)\n value double precision,\n metadata jsonb DEFAULT '{}',\n\n -- Primary key last (for TOAST ordering)\n id uuid DEFAULT uuidv7(),\n\n PRIMARY KEY (recorded_at, device_id, id)\n) WITH (fillfactor = 90); -- Allow for some updates\n```\n\n## Partitioning for Time-Series\n\n### Monthly Partitioning\n\n```sql\n-- Create partitioned table\nCREATE TABLE data.events (\n id uuid NOT NULL DEFAULT uuidv7(),\n event_type text NOT NULL,\n device_id uuid NOT NULL,\n value double precision,\n recorded_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (id, recorded_at)\n) PARTITION BY RANGE (recorded_at);\n\n-- Create monthly partitions\nCREATE TABLE data.events_2024_01 PARTITION OF data.events\n FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');\nCREATE TABLE data.events_2024_02 PARTITION OF data.events\n FOR VALUES FROM ('2024-02-01') TO ('2024-03-01');\n-- ... continue for each month\n\n-- Default partition for out-of-range data\nCREATE TABLE data.events_default PARTITION OF data.events DEFAULT;\n```\n\n### Daily Partitioning (High Volume)\n\n```sql\n-- For very high volume data\nCREATE TABLE data.logs (\n id uuid NOT NULL DEFAULT uuidv7(),\n level text NOT NULL,\n message text NOT NULL,\n recorded_at timestamptz NOT NULL DEFAULT now(),\n\n PRIMARY KEY (id, recorded_at)\n) PARTITION BY RANGE (recorded_at);\n\n-- Automated daily partition creation\nCREATE FUNCTION private.create_daily_partition(\n in_table_name text,\n in_date date\n)\nRETURNS text\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_partition_name text;\n l_start_date date := in_date;\n l_end_date date := in_date + 1;\nBEGIN\n l_partition_name := in_table_name || '_' || to_char(in_date, 'YYYY_MM_DD');\n\n EXECUTE format(\n 'CREATE TABLE IF NOT EXISTS data.%I PARTITION OF data.%I FOR VALUES FROM (%L) TO (%L)',\n l_partition_name, in_table_name, l_start_date, l_end_date\n );\n\n RETURN l_partition_name;\nEND;\n$;\n\n-- Create next 7 days of partitions\nSELECT private.create_daily_partition('logs', current_date + i)\nFROM generate_series(0, 7) AS i;\n```\n\n### Partition Maintenance Automation\n\n```sql\n-- Create upcoming partitions and drop old ones\nCREATE PROCEDURE private.maintain_time_series_partitions(\n in_table_name text,\n in_partition_interval interval,\n in_create_ahead integer,\n in_retention_periods integer\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_current_period date;\n l_i integer;\nBEGIN\n -- Create upcoming partitions\n FOR l_i IN 0..in_create_ahead LOOP\n l_current_period := date_trunc(\n CASE\n WHEN in_partition_interval = '1 month' THEN 'month'\n WHEN in_partition_interval = '1 day' THEN 'day'\n WHEN in_partition_interval = '1 week' THEN 'week'\n END,\n now() + (l_i * in_partition_interval)\n );\n\n PERFORM private.create_partition(in_table_name, l_current_period, in_partition_interval);\n END LOOP;\n\n -- Drop old partitions\n PERFORM private.drop_old_partitions(\n in_table_name,\n now() - (in_retention_periods * in_partition_interval)\n );\nEND;\n$;\n\n-- Schedule with pg_cron\nSELECT cron.schedule(\n 'maintain-events-partitions',\n '0 1 * * *',\n $CALL private.maintain_time_series_partitions('events', '1 month', 3, 12)$\n);\n```\n\n## Indexing Strategies\n\n### BRIN Index (Block Range Index)\n\n```sql\n-- BRIN: Perfect for naturally time-ordered data\n-- Very small index size, fast range scans\n\nCREATE INDEX events_recorded_at_brin_idx\n ON data.events USING brin (recorded_at)\n WITH (pages_per_range = 128);\n\n-- BRIN works best when data is physically ordered\n-- Insert data in time order for best results\n\n-- Check correlation (should be close to 1 or -1)\nSELECT correlation\nFROM pg_stats\nWHERE tablename = 'events' AND attname = 'recorded_at';\n```\n\n### Composite B-tree Index\n\n```sql\n-- For queries filtering by device + time range\nCREATE INDEX events_device_time_idx\n ON data.events (device_id, recorded_at DESC);\n\n-- Covering index for common queries\nCREATE INDEX events_device_time_covering_idx\n ON data.events (device_id, recorded_at DESC)\n INCLUDE (event_type, value);\n```\n\n### Partial Indexes for Recent Data\n\n```sql\n-- Index only recent data (hot data)\nCREATE INDEX events_recent_idx\n ON data.events (device_id, recorded_at)\n WHERE recorded_at > now() - interval '7 days';\n\n-- Recreate periodically to update the condition\n-- This requires a function to manage the index\n\nCREATE PROCEDURE private.refresh_recent_index()\nLANGUAGE plpgsql\nAS $\nBEGIN\n DROP INDEX IF EXISTS data.events_recent_idx;\n\n EXECUTE format(\n 'CREATE INDEX events_recent_idx ON data.events (device_id, recorded_at) WHERE recorded_at > %L',\n now() - interval '7 days'\n );\nEND;\n$;\n```\n\n### Per-Partition Indexes\n\n```sql\n-- Indexes are automatically created on each partition\n-- when you create an index on the parent table\n\nCREATE INDEX events_type_idx ON data.events (event_type);\n-- Creates: events_2024_01_event_type_idx, events_2024_02_event_type_idx, etc.\n\n-- For special per-partition indexes\nCREATE INDEX events_2024_01_special_idx\n ON data.events_2024_01 (value)\n WHERE event_type = 'critical';\n```\n\n## Query Patterns\n\n### Time Range Query\n\n```sql\n-- Basic range query (uses partition pruning)\nSELECT *\nFROM data.events\nWHERE recorded_at >= '2024-03-01'\n AND recorded_at \u003c '2024-03-02'\nORDER BY recorded_at;\n\n-- With device filter\nSELECT *\nFROM data.events\nWHERE device_id = 'device-uuid'\n AND recorded_at >= now() - interval '24 hours'\nORDER BY recorded_at DESC\nLIMIT 100;\n```\n\n### Latest Value per Device\n\n```sql\n-- Get most recent reading per device\nSELECT DISTINCT ON (device_id)\n device_id,\n recorded_at,\n value\nFROM data.events\nWHERE recorded_at >= now() - interval '1 hour'\nORDER BY device_id, recorded_at DESC;\n\n-- Alternative using lateral join (can be faster)\nSELECT d.id AS device_id, e.recorded_at, e.value\nFROM data.devices d\nCROSS JOIN LATERAL (\n SELECT recorded_at, value\n FROM data.events\n WHERE device_id = d.id\n ORDER BY recorded_at DESC\n LIMIT 1\n) e;\n```\n\n### Time Bucketing\n\n```sql\n-- Aggregate into time buckets\nSELECT\n date_trunc('hour', recorded_at) AS bucket,\n device_id,\n avg(value) AS avg_value,\n min(value) AS min_value,\n max(value) AS max_value,\n count(*) AS count\nFROM data.events\nWHERE recorded_at >= now() - interval '24 hours'\nGROUP BY 1, 2\nORDER BY 1 DESC, 2;\n\n-- Custom bucket size (15 minutes)\nSELECT\n date_bin('15 minutes', recorded_at, timestamptz '2024-01-01') AS bucket,\n device_id,\n avg(value) AS avg_value\nFROM data.events\nWHERE recorded_at >= now() - interval '24 hours'\nGROUP BY 1, 2\nORDER BY 1;\n```\n\n### Gap Detection\n\n```sql\n-- Find gaps in time-series data\nWITH readings AS (\n SELECT\n device_id,\n recorded_at,\n lead(recorded_at) OVER (PARTITION BY device_id ORDER BY recorded_at) AS next_at\n FROM data.events\n WHERE device_id = 'device-uuid'\n AND recorded_at >= now() - interval '24 hours'\n)\nSELECT\n device_id,\n recorded_at AS gap_start,\n next_at AS gap_end,\n next_at - recorded_at AS gap_duration\nFROM readings\nWHERE next_at - recorded_at > interval '5 minutes'\nORDER BY gap_duration DESC;\n```\n\n### Moving Averages\n\n```sql\n-- Rolling average over last N readings\nSELECT\n recorded_at,\n value,\n avg(value) OVER (\n ORDER BY recorded_at\n ROWS BETWEEN 9 PRECEDING AND CURRENT ROW\n ) AS moving_avg_10,\n avg(value) OVER (\n ORDER BY recorded_at\n ROWS BETWEEN 59 PRECEDING AND CURRENT ROW\n ) AS moving_avg_60\nFROM data.events\nWHERE device_id = 'device-uuid'\n AND recorded_at >= now() - interval '1 hour'\nORDER BY recorded_at;\n\n-- Time-based window (last 5 minutes)\nSELECT\n recorded_at,\n value,\n avg(value) OVER (\n ORDER BY recorded_at\n RANGE BETWEEN interval '5 minutes' PRECEDING AND CURRENT ROW\n ) AS moving_avg\nFROM data.events\nWHERE device_id = 'device-uuid'\nORDER BY recorded_at;\n```\n\n## Downsampling & Aggregation\n\n### Continuous Aggregation (Materialized View)\n\n```sql\n-- Create materialized view for hourly aggregates\nCREATE MATERIALIZED VIEW data.mv_events_hourly AS\nSELECT\n date_trunc('hour', recorded_at) AS bucket,\n device_id,\n event_type,\n count(*) AS event_count,\n avg(value) AS avg_value,\n min(value) AS min_value,\n max(value) AS max_value,\n percentile_cont(0.95) WITHIN GROUP (ORDER BY value) AS p95_value\nFROM data.events\nGROUP BY 1, 2, 3\nWITH NO DATA;\n\n-- Create indexes on materialized view\nCREATE INDEX mv_events_hourly_bucket_idx ON data.mv_events_hourly (bucket);\nCREATE INDEX mv_events_hourly_device_idx ON data.mv_events_hourly (device_id, bucket);\n\n-- Initial population\nREFRESH MATERIALIZED VIEW data.mv_events_hourly;\n\n-- Schedule refresh\nSELECT cron.schedule(\n 'refresh-events-hourly',\n '5 * * * *', -- 5 minutes after each hour\n $REFRESH MATERIALIZED VIEW CONCURRENTLY data.mv_events_hourly$\n);\n```\n\n### Incremental Aggregation Table\n\n```sql\n-- Pre-computed aggregates table\nCREATE TABLE data.events_hourly (\n bucket timestamptz NOT NULL,\n device_id uuid NOT NULL,\n event_type text NOT NULL,\n event_count integer NOT NULL DEFAULT 0,\n value_sum double precision NOT NULL DEFAULT 0,\n value_min double precision,\n value_max double precision,\n\n PRIMARY KEY (bucket, device_id, event_type)\n);\n\n-- Upsert aggregates\nCREATE PROCEDURE private.aggregate_events_hourly(in_hour timestamptz)\nLANGUAGE plpgsql\nAS $\nBEGIN\n INSERT INTO data.events_hourly (bucket, device_id, event_type, event_count, value_sum, value_min, value_max)\n SELECT\n date_trunc('hour', recorded_at) AS bucket,\n device_id,\n event_type,\n count(*),\n sum(value),\n min(value),\n max(value)\n FROM data.events\n WHERE recorded_at >= in_hour\n AND recorded_at \u003c in_hour + interval '1 hour'\n GROUP BY 1, 2, 3\n ON CONFLICT (bucket, device_id, event_type) DO UPDATE SET\n event_count = EXCLUDED.event_count,\n value_sum = EXCLUDED.value_sum,\n value_min = LEAST(events_hourly.value_min, EXCLUDED.value_min),\n value_max = GREATEST(events_hourly.value_max, EXCLUDED.value_max);\nEND;\n$;\n\n-- Schedule aggregation\nSELECT cron.schedule(\n 'aggregate-events-hourly',\n '0 * * * *',\n $CALL private.aggregate_events_hourly(date_trunc('hour', now() - interval '1 hour'))$\n);\n```\n\n### Multi-Resolution Aggregates\n\n```sql\n-- Store aggregates at multiple resolutions\nCREATE TABLE data.events_1min (\n bucket timestamptz, device_id uuid, event_count int, value_avg double precision,\n PRIMARY KEY (bucket, device_id)\n);\n\nCREATE TABLE data.events_5min (\n bucket timestamptz, device_id uuid, event_count int, value_avg double precision,\n PRIMARY KEY (bucket, device_id)\n);\n\nCREATE TABLE data.events_1hour (\n bucket timestamptz, device_id uuid, event_count int, value_avg double precision,\n PRIMARY KEY (bucket, device_id)\n);\n\n-- Query function that selects appropriate resolution\nCREATE FUNCTION api.get_events_aggregated(\n in_device_id uuid,\n in_start timestamptz,\n in_end timestamptz\n)\nRETURNS TABLE (bucket timestamptz, event_count int, value_avg double precision)\nLANGUAGE plpgsql\nSTABLE\nAS $\nDECLARE\n l_duration interval := in_end - in_start;\nBEGIN\n IF l_duration \u003c= interval '1 hour' THEN\n -- Use raw data for short ranges\n RETURN QUERY\n SELECT\n date_trunc('minute', recorded_at),\n count(*)::int,\n avg(value)\n FROM data.events\n WHERE device_id = in_device_id\n AND recorded_at >= in_start AND recorded_at \u003c in_end\n GROUP BY 1;\n ELSIF l_duration \u003c= interval '1 day' THEN\n -- Use 5-minute aggregates\n RETURN QUERY\n SELECT bucket, event_count, value_avg\n FROM data.events_5min\n WHERE device_id = in_device_id\n AND bucket >= in_start AND bucket \u003c in_end;\n ELSE\n -- Use hourly aggregates\n RETURN QUERY\n SELECT bucket, event_count, value_avg\n FROM data.events_1hour\n WHERE device_id = in_device_id\n AND bucket >= in_start AND bucket \u003c in_end;\n END IF;\nEND;\n$;\n```\n\n## Retention Management\n\n### Drop Old Partitions\n\n```sql\n-- Drop partitions older than retention period\nCREATE FUNCTION private.drop_old_partitions(\n in_parent_table text,\n in_cutoff timestamptz\n)\nRETURNS integer\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_partition record;\n l_count integer := 0;\nBEGIN\n FOR l_partition IN\n SELECT c.relname AS partition_name\n FROM pg_inherits i\n JOIN pg_class p ON i.inhparent = p.oid\n JOIN pg_class c ON i.inhrelid = c.oid\n WHERE p.relname = in_parent_table\n LOOP\n -- Extract date from partition name and compare\n -- Assumes naming like events_2024_01\n BEGIN\n IF to_date(\n substring(l_partition.partition_name from '\\d{4}_\\d{2}'),\n 'YYYY_MM'\n ) \u003c date_trunc('month', in_cutoff) THEN\n EXECUTE format('DROP TABLE data.%I', l_partition.partition_name);\n l_count := l_count + 1;\n RAISE NOTICE 'Dropped partition: %', l_partition.partition_name;\n END IF;\n EXCEPTION WHEN OTHERS THEN\n -- Skip partitions that don't match expected format\n NULL;\n END;\n END LOOP;\n\n RETURN l_count;\nEND;\n$;\n```\n\n### Archive Before Drop\n\n```sql\n-- Archive to cold storage before dropping\nCREATE PROCEDURE private.archive_and_drop_partition(\n in_partition_name text,\n in_archive_path text\n)\nLANGUAGE plpgsql\nAS $\nBEGIN\n -- Export to CSV\n EXECUTE format(\n 'COPY data.%I TO %L WITH (FORMAT CSV, HEADER)',\n in_partition_name,\n in_archive_path || '/' || in_partition_name || '.csv'\n );\n\n -- Drop after successful export\n EXECUTE format('DROP TABLE data.%I', in_partition_name);\n\n RAISE NOTICE 'Archived and dropped: %', in_partition_name;\nEND;\n$;\n```\n\n## Performance Optimization\n\n### Bulk Insert Optimization\n\n```sql\n-- Batch insert with COPY\nCOPY data.events (device_id, event_type, value, recorded_at)\nFROM '/path/to/data.csv' WITH (FORMAT CSV, HEADER);\n\n-- Or use COPY ... FROM STDIN in application\n-- COPY data.events FROM STDIN WITH (FORMAT CSV);\n\n-- For streaming inserts, use prepared statements\nPREPARE insert_event AS\n INSERT INTO data.events (device_id, event_type, value, recorded_at)\n VALUES ($1, $2, $3, $4);\n```\n\n### Query Performance Tips\n\n```sql\n-- 1. Always include time range in WHERE clause\n-- ❌ Bad: Full table scan\nSELECT * FROM data.events WHERE device_id = 'uuid';\n\n-- ✅ Good: Partition pruning\nSELECT * FROM data.events\nWHERE device_id = 'uuid'\n AND recorded_at >= now() - interval '24 hours';\n\n-- 2. Use LIMIT for recent data queries\nSELECT * FROM data.events\nWHERE recorded_at >= now() - interval '1 hour'\nORDER BY recorded_at DESC\nLIMIT 1000;\n\n-- 3. Analyze tables after bulk loads\nANALYZE data.events;\n\n-- 4. Check execution plan\nEXPLAIN (ANALYZE, BUFFERS)\nSELECT * FROM data.events\nWHERE device_id = 'uuid'\n AND recorded_at >= '2024-03-01'\n AND recorded_at \u003c '2024-03-02';\n```\n\n### Configuration Tuning\n\n```sql\n-- postgresql.conf settings for time-series\n\n-- Increase for write-heavy workloads\ncheckpoint_completion_target = 0.9\nwal_buffers = 64MB\n\n-- For large time range queries\nwork_mem = 256MB -- Per operation\neffective_cache_size = 12GB -- 75% of RAM\n\n-- For BRIN indexes\neffective_io_concurrency = 200 -- For SSD\nrandom_page_cost = 1.1 -- For SSD\n\n-- Autovacuum tuning for append-only tables\nALTER TABLE data.events SET (\n autovacuum_vacuum_scale_factor = 0,\n autovacuum_vacuum_threshold = 10000,\n autovacuum_analyze_scale_factor = 0,\n autovacuum_analyze_threshold = 10000\n);\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":19246,"content_sha256":"c2d04ce1aae396e34bedd03899eae22c46c0edfeabd413630ddd1d1e46b7405f"},{"filename":"references/transaction-patterns.md","content":"# Advanced Transaction Patterns\n\nThis document covers PostgreSQL transaction management including isolation levels, locking strategies, deadlock prevention, and savepoints.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Isolation Levels](#isolation-levels)\n3. [Locking Strategies](#locking-strategies)\n4. [Advisory Locks](#advisory-locks)\n5. [Deadlock Prevention](#deadlock-prevention)\n6. [Savepoints](#savepoints)\n7. [Long-Running Transactions](#long-running-transactions)\n8. [Transaction Patterns](#transaction-patterns)\n\n## Overview\n\n### Transaction Properties (ACID)\n\n| Property | Meaning | PostgreSQL Support |\n|----------|---------|-------------------|\n| **A**tomicity | All or nothing | ✅ Full |\n| **C**onsistency | Valid state to valid state | ✅ Constraints |\n| **I**solation | Concurrent transactions isolated | ✅ 4 levels |\n| **D**urability | Committed = permanent | ✅ WAL |\n\n### Transaction Decision Tree\n\n```mermaid\nflowchart TD\n START([Transaction Need]) --> Q1{Read or Write?}\n\n Q1 -->|Read only| Q2{Need consistent snapshot?}\n Q1 -->|Write| Q3{Concurrent modification risk?}\n\n Q2 -->|Yes| REPEATABLE[\"REPEATABLE READ\u003cbr/>Snapshot at start\"]\n Q2 -->|No| READ_COMMITTED[\"READ COMMITTED\u003cbr/>(Default)\"]\n\n Q3 -->|High| Q4{Can retry?}\n Q3 -->|Low| READ_COMMITTED\n\n Q4 -->|Yes| SERIALIZABLE[\"SERIALIZABLE\u003cbr/>+ Retry logic\"]\n Q4 -->|No| LOCKING[\"Explicit locking\u003cbr/>SELECT FOR UPDATE\"]\n\n style SERIALIZABLE fill:#fff3e0\n style LOCKING fill:#bbdefb\n```\n\n## Isolation Levels\n\n### Isolation Level Comparison\n\n| Phenomenon | Read Uncommitted | Read Committed | Repeatable Read | Serializable |\n|------------|-----------------|----------------|-----------------|--------------|\n| Dirty read | Possible | ❌ No | ❌ No | ❌ No |\n| Non-repeatable read | Possible | Possible | ❌ No | ❌ No |\n| Phantom read | Possible | Possible | ❌ No* | ❌ No |\n| Serialization anomaly | Possible | Possible | Possible | ❌ No |\n\n*PostgreSQL's Repeatable Read prevents phantoms (uses MVCC).\n\n### Read Committed (Default)\n\n```sql\n-- Default isolation level\n-- Each statement sees committed data as of statement start\nBEGIN;\nSET TRANSACTION ISOLATION LEVEL READ COMMITTED;\n\nSELECT * FROM data.accounts WHERE id = 1;\n-- Another transaction commits change to account 1\nSELECT * FROM data.accounts WHERE id = 1; -- Sees new value!\n\nCOMMIT;\n```\n\n### Repeatable Read\n\n```sql\n-- Snapshot at transaction start\n-- Same query always returns same results within transaction\nBEGIN;\nSET TRANSACTION ISOLATION LEVEL REPEATABLE READ;\n\nSELECT * FROM data.accounts WHERE id = 1; -- Sees balance = 100\n-- Another transaction commits: balance = 50\nSELECT * FROM data.accounts WHERE id = 1; -- Still sees balance = 100!\n\n-- But: Write conflict causes error\nUPDATE data.accounts SET balance = balance - 10 WHERE id = 1;\n-- ERROR: could not serialize access due to concurrent update\n\nROLLBACK;\n```\n\n### Serializable\n\n```sql\n-- Full serializability - as if transactions ran sequentially\n-- May cause serialization_failure errors - must retry\nBEGIN;\nSET TRANSACTION ISOLATION LEVEL SERIALIZABLE;\n\n-- Transaction 1: Sum and insert\nSELECT SUM(balance) FROM data.accounts;\nINSERT INTO data.totals (amount) VALUES (sum_result);\n\n-- Transaction 2: Same operations concurrently\n-- One will succeed, other gets serialization_failure\n\nCOMMIT;\n```\n\n### Set Isolation Level\n\n```sql\n-- Per transaction\nBEGIN;\nSET TRANSACTION ISOLATION LEVEL SERIALIZABLE;\n-- Or\nBEGIN ISOLATION LEVEL SERIALIZABLE;\n\n-- Per session (default for all transactions)\nSET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL REPEATABLE READ;\n\n-- Check current level\nSHOW transaction_isolation;\n```\n\n## Locking Strategies\n\n### Row-Level Locks\n\n```sql\n-- FOR UPDATE: Exclusive lock, blocks other FOR UPDATE and modifications\nSELECT * FROM data.accounts WHERE id = 1 FOR UPDATE;\n\n-- FOR NO KEY UPDATE: Like FOR UPDATE but allows FOR KEY SHARE\nSELECT * FROM data.accounts WHERE id = 1 FOR NO KEY UPDATE;\n\n-- FOR SHARE: Shared lock, blocks modifications but allows other FOR SHARE\nSELECT * FROM data.accounts WHERE id = 1 FOR SHARE;\n\n-- FOR KEY SHARE: Weakest lock, blocks only modifications to key columns\nSELECT * FROM data.accounts WHERE id = 1 FOR KEY SHARE;\n```\n\n### NOWAIT and SKIP LOCKED\n\n```sql\n-- NOWAIT: Fail immediately if lock not available\nBEGIN;\nSELECT * FROM data.accounts WHERE id = 1 FOR UPDATE NOWAIT;\n-- ERROR: could not obtain lock on row in relation \"accounts\"\nROLLBACK;\n\n-- SKIP LOCKED: Skip locked rows (perfect for job queues)\nSELECT * FROM data.jobs\nWHERE status = 'pending'\nORDER BY created_at\nLIMIT 1\nFOR UPDATE SKIP LOCKED;\n```\n\n### Table-Level Locks\n\n```sql\n-- Explicit table lock (rarely needed)\nLOCK TABLE data.accounts IN SHARE MODE;\nLOCK TABLE data.accounts IN EXCLUSIVE MODE;\n\n-- ACCESS SHARE (SELECT): Conflicts only with ACCESS EXCLUSIVE\n-- ROW SHARE (SELECT FOR UPDATE/SHARE): Conflicts with EXCLUSIVE\n-- ROW EXCLUSIVE (UPDATE, DELETE, INSERT): Conflicts with SHARE, EXCLUSIVE\n-- SHARE: Conflicts with ROW EXCLUSIVE\n-- EXCLUSIVE: Conflicts with everything except ACCESS SHARE\n-- ACCESS EXCLUSIVE (ALTER TABLE, DROP): Conflicts with everything\n```\n\n### Optimistic Locking Pattern\n\n```sql\n-- Add version column\nALTER TABLE data.orders ADD COLUMN version integer NOT NULL DEFAULT 1;\n\n-- Update with version check\nCREATE PROCEDURE api.update_order_optimistic(\n in_order_id uuid,\n in_expected_version integer,\n in_status text,\n INOUT io_new_version integer DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_updated integer;\nBEGIN\n UPDATE data.orders\n SET status = in_status,\n version = version + 1,\n updated_at = now()\n WHERE id = in_order_id\n AND version = in_expected_version\n RETURNING version INTO io_new_version;\n\n GET DIAGNOSTICS l_updated = ROW_COUNT;\n\n IF l_updated = 0 THEN\n -- Check if exists\n IF EXISTS (SELECT 1 FROM data.orders WHERE id = in_order_id) THEN\n RAISE EXCEPTION 'Concurrent modification detected'\n USING ERRCODE = 'P0002';\n ELSE\n RAISE EXCEPTION 'Order not found: %', in_order_id\n USING ERRCODE = 'P0002';\n END IF;\n END IF;\nEND;\n$;\n```\n\n### Pessimistic Locking Pattern\n\n```sql\n-- Lock row for entire transaction\nCREATE PROCEDURE api.transfer_funds(\n in_from_account uuid,\n in_to_account uuid,\n in_amount numeric\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_from_balance numeric;\nBEGIN\n -- Lock both accounts in consistent order (prevents deadlock)\n -- Always lock by sorted ID\n IF in_from_account \u003c in_to_account THEN\n PERFORM 1 FROM data.accounts WHERE id = in_from_account FOR UPDATE;\n PERFORM 1 FROM data.accounts WHERE id = in_to_account FOR UPDATE;\n ELSE\n PERFORM 1 FROM data.accounts WHERE id = in_to_account FOR UPDATE;\n PERFORM 1 FROM data.accounts WHERE id = in_from_account FOR UPDATE;\n END IF;\n\n -- Check balance\n SELECT balance INTO l_from_balance\n FROM data.accounts WHERE id = in_from_account;\n\n IF l_from_balance \u003c in_amount THEN\n RAISE EXCEPTION 'Insufficient funds'\n USING ERRCODE = 'P0001';\n END IF;\n\n -- Perform transfer\n UPDATE data.accounts SET balance = balance - in_amount\n WHERE id = in_from_account;\n\n UPDATE data.accounts SET balance = balance + in_amount\n WHERE id = in_to_account;\nEND;\n$;\n```\n\n## Advisory Locks\n\n### Session-Level Advisory Locks\n\n```sql\n-- Acquire lock (blocks until available)\nSELECT pg_advisory_lock(12345);\n\n-- Try to acquire (returns immediately)\nSELECT pg_try_advisory_lock(12345); -- Returns true/false\n\n-- Release lock\nSELECT pg_advisory_unlock(12345);\n\n-- Two-argument form (for namespacing)\nSELECT pg_advisory_lock(1, 100); -- Namespace 1, ID 100\n```\n\n### Transaction-Level Advisory Locks\n\n```sql\n-- Automatically released at transaction end\nBEGIN;\nSELECT pg_advisory_xact_lock(12345);\n-- Do work...\nCOMMIT; -- Lock automatically released\n```\n\n### Advisory Lock Patterns\n\n```sql\n-- Prevent concurrent execution of a function\nCREATE FUNCTION api.singleton_task()\nRETURNS void\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n co_lock_id constant bigint := 999999;\nBEGIN\n -- Try to get exclusive lock\n IF NOT pg_try_advisory_lock(co_lock_id) THEN\n RAISE NOTICE 'Task already running, skipping';\n RETURN;\n END IF;\n\n BEGIN\n -- Do exclusive work\n RAISE NOTICE 'Running singleton task';\n PERFORM pg_sleep(5);\n EXCEPTION WHEN OTHERS THEN\n -- Ensure lock is released on error\n PERFORM pg_advisory_unlock(co_lock_id);\n RAISE;\n END;\n\n PERFORM pg_advisory_unlock(co_lock_id);\nEND;\n$;\n\n-- Entity-specific lock\nCREATE FUNCTION private.get_lock_id(in_entity_type text, in_entity_id uuid)\nRETURNS bigint\nLANGUAGE sql\nIMMUTABLE\nAS $\n -- Generate consistent lock ID from entity\n SELECT abs(hashtext(in_entity_type || ':' || in_entity_id::text))::bigint;\n$;\n\n-- Use for entity-level locking\nSELECT pg_advisory_xact_lock(private.get_lock_id('order', 'order-uuid'));\n```\n\n## Deadlock Prevention\n\n### Deadlock Detection\n\n```sql\n-- PostgreSQL automatically detects deadlocks\n-- One transaction is aborted with:\n-- ERROR: deadlock detected\n-- DETAIL: Process 12345 waits for ShareLock on transaction 67890; blocked by process 67891.\n\n-- Configure deadlock detection timeout\nSET deadlock_timeout = '1s'; -- Default\n```\n\n### Prevention Strategies\n\n```sql\n-- 1. Always lock resources in same order\nCREATE PROCEDURE api.update_related_entities(\n in_entity_ids uuid[]\n)\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_sorted_ids uuid[];\nBEGIN\n -- Sort IDs for consistent lock order\n SELECT array_agg(id ORDER BY id) INTO l_sorted_ids\n FROM unnest(in_entity_ids) AS id;\n\n -- Lock in sorted order\n PERFORM 1 FROM data.entities\n WHERE id = ANY(l_sorted_ids)\n ORDER BY id\n FOR UPDATE;\n\n -- Now safe to update\n -- ...\nEND;\n$;\n\n-- 2. Use NOWAIT with retry\nCREATE FUNCTION private.acquire_lock_with_retry(\n in_table text,\n in_id uuid,\n in_max_attempts integer DEFAULT 3\n)\nRETURNS boolean\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_attempt integer := 0;\nBEGIN\n LOOP\n l_attempt := l_attempt + 1;\n\n BEGIN\n EXECUTE format(\n 'SELECT 1 FROM %I WHERE id = $1 FOR UPDATE NOWAIT',\n in_table\n ) USING in_id;\n RETURN true;\n EXCEPTION\n WHEN lock_not_available THEN\n IF l_attempt >= in_max_attempts THEN\n RETURN false;\n END IF;\n PERFORM pg_sleep(0.1 * l_attempt); -- Exponential backoff\n END;\n END LOOP;\nEND;\n$;\n\n-- 3. Reduce lock scope\n-- BAD: Lock entire table\nLOCK TABLE data.accounts;\n\n-- GOOD: Lock only needed rows\nSELECT 1 FROM data.accounts WHERE id = in_id FOR UPDATE;\n```\n\n### Monitor for Deadlocks\n\n```sql\n-- Check for blocked queries\nSELECT\n blocked.pid AS blocked_pid,\n blocked.usename AS blocked_user,\n blocked.query AS blocked_query,\n blocking.pid AS blocking_pid,\n blocking.usename AS blocking_user,\n blocking.query AS blocking_query\nFROM pg_stat_activity AS blocked\nJOIN pg_locks AS blocked_locks ON blocked.pid = blocked_locks.pid\nJOIN pg_locks AS blocking_locks ON blocked_locks.locktype = blocking_locks.locktype\n AND blocked_locks.database IS NOT DISTINCT FROM blocking_locks.database\n AND blocked_locks.relation IS NOT DISTINCT FROM blocking_locks.relation\n AND blocked_locks.page IS NOT DISTINCT FROM blocking_locks.page\n AND blocked_locks.tuple IS NOT DISTINCT FROM blocking_locks.tuple\n AND blocked_locks.transactionid IS NOT DISTINCT FROM blocking_locks.transactionid\n AND blocked_locks.classid IS NOT DISTINCT FROM blocking_locks.classid\n AND blocked_locks.objid IS NOT DISTINCT FROM blocking_locks.objid\n AND blocked_locks.objsubid IS NOT DISTINCT FROM blocking_locks.objsubid\n AND blocked_locks.pid != blocking_locks.pid\nJOIN pg_stat_activity AS blocking ON blocking.pid = blocking_locks.pid\nWHERE NOT blocked_locks.granted;\n\n-- Deadlock statistics\nSELECT * FROM pg_stat_database WHERE datname = current_database();\n-- Check: deadlocks column\n```\n\n## Savepoints\n\n### Basic Savepoints\n\n```sql\nBEGIN;\n\nINSERT INTO data.orders (customer_id) VALUES ('cust-1');\nSAVEPOINT order_created;\n\nINSERT INTO data.order_items (order_id, product_id) VALUES ('order-1', 'prod-1');\nSAVEPOINT items_added;\n\n-- Something goes wrong\nINSERT INTO data.order_items (order_id, product_id) VALUES ('order-1', 'invalid');\n-- ERROR!\n\n-- Rollback to savepoint (not entire transaction)\nROLLBACK TO SAVEPOINT items_added;\n\n-- Continue with valid data\nINSERT INTO data.order_items (order_id, product_id) VALUES ('order-1', 'prod-2');\n\nCOMMIT; -- Order and valid items are committed\n```\n\n### Savepoint Pattern for Batch Processing\n\n```sql\nCREATE PROCEDURE api.process_batch(in_items jsonb)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nDECLARE\n l_item jsonb;\n l_success_count integer := 0;\n l_error_count integer := 0;\nBEGIN\n FOR l_item IN SELECT * FROM jsonb_array_elements(in_items)\n LOOP\n -- Savepoint for each item\n BEGIN\n SAVEPOINT process_item;\n\n -- Process item\n INSERT INTO data.processed_items (data)\n VALUES (l_item);\n\n l_success_count := l_success_count + 1;\n\n EXCEPTION WHEN OTHERS THEN\n -- Rollback just this item\n ROLLBACK TO SAVEPOINT process_item;\n l_error_count := l_error_count + 1;\n\n -- Log error\n INSERT INTO data.processing_errors (item_data, error_message)\n VALUES (l_item, SQLERRM);\n END;\n END LOOP;\n\n RAISE NOTICE 'Processed: % success, % errors', l_success_count, l_error_count;\nEND;\n$;\n```\n\n### Nested Transaction Simulation\n\n```sql\n-- PostgreSQL doesn't have true nested transactions\n-- Use savepoints to simulate\n\nCREATE PROCEDURE api.outer_procedure()\nLANGUAGE plpgsql\nAS $\nBEGIN\n INSERT INTO data.log (message) VALUES ('Outer started');\n\n -- \"Nested transaction\" via savepoint\n BEGIN\n SAVEPOINT nested;\n CALL private.inner_procedure();\n EXCEPTION WHEN OTHERS THEN\n ROLLBACK TO SAVEPOINT nested;\n INSERT INTO data.log (message) VALUES ('Inner failed, continuing');\n END;\n\n INSERT INTO data.log (message) VALUES ('Outer completed');\nEND;\n$;\n```\n\n## Long-Running Transactions\n\n### Problems with Long Transactions\n\n```sql\n-- Long transactions cause:\n-- 1. Lock contention\n-- 2. MVCC bloat (old versions retained)\n-- 3. Replication lag\n-- 4. Autovacuum blocked\n```\n\n### Chunked Processing Pattern\n\n```sql\nCREATE PROCEDURE api.process_large_table()\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_batch_size constant integer := 1000;\n l_last_id uuid := '00000000-0000-0000-0000-000000000000';\n l_processed integer;\nBEGIN\n LOOP\n -- Process one batch in its own transaction\n WITH batch AS (\n SELECT id\n FROM data.large_table\n WHERE id > l_last_id\n AND processed = false\n ORDER BY id\n LIMIT l_batch_size\n FOR UPDATE SKIP LOCKED\n )\n UPDATE data.large_table t\n SET processed = true\n FROM batch\n WHERE t.id = batch.id\n RETURNING t.id INTO l_last_id;\n\n GET DIAGNOSTICS l_processed = ROW_COUNT;\n\n -- Exit when no more rows\n EXIT WHEN l_processed = 0;\n\n -- Commit batch (implicit in procedure)\n COMMIT;\n\n RAISE NOTICE 'Processed batch, last_id: %', l_last_id;\n\n -- Optional: Small delay to reduce load\n PERFORM pg_sleep(0.1);\n END LOOP;\nEND;\n$;\n```\n\n### Monitor Long Transactions\n\n```sql\n-- Find long-running transactions\nSELECT\n pid,\n now() - xact_start AS duration,\n state,\n query\nFROM pg_stat_activity\nWHERE xact_start IS NOT NULL\n AND state != 'idle'\nORDER BY xact_start;\n\n-- Kill long transaction (if needed)\nSELECT pg_terminate_backend(pid);\n```\n\n## Transaction Patterns\n\n### Retry Pattern for Serialization Failures\n\n```sql\nCREATE FUNCTION api.execute_with_retry(\n in_sql text,\n in_max_retries integer DEFAULT 3\n)\nRETURNS void\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_retry integer := 0;\nBEGIN\n LOOP\n BEGIN\n EXECUTE in_sql;\n RETURN; -- Success\n EXCEPTION\n WHEN serialization_failure OR deadlock_detected THEN\n l_retry := l_retry + 1;\n IF l_retry > in_max_retries THEN\n RAISE; -- Give up\n END IF;\n RAISE NOTICE 'Retry % after %', l_retry, SQLERRM;\n PERFORM pg_sleep(0.1 * power(2, l_retry)); -- Exponential backoff\n END;\n END LOOP;\nEND;\n$;\n```\n\n### Two-Phase Commit (for distributed transactions)\n\n```sql\n-- Prepare transaction (rarely used, for distributed systems)\nBEGIN;\n-- Do work\nPREPARE TRANSACTION 'my_transaction_id';\n\n-- Later, commit or rollback\nCOMMIT PREPARED 'my_transaction_id';\n-- Or\nROLLBACK PREPARED 'my_transaction_id';\n\n-- View prepared transactions\nSELECT * FROM pg_prepared_xacts;\n```\n\n### Read-Only Transaction\n\n```sql\n-- Explicitly read-only (optimization hint)\nBEGIN READ ONLY;\nSELECT * FROM data.accounts;\n-- UPDATE would fail: ERROR: cannot execute UPDATE in a read-only transaction\nCOMMIT;\n\n-- Deferrable (can wait for clean snapshot in serializable)\nBEGIN ISOLATION LEVEL SERIALIZABLE READ ONLY DEFERRABLE;\n-- Good for long-running reports\nCOMMIT;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":17689,"content_sha256":"1184220a88786fcea40615299175f40ae4bba8976638504e8ad6bb7df85197d3"},{"filename":"references/vector-search.md","content":"# Vector Search with pgvector\n\nThis document covers vector similarity search using the pgvector extension, including embedding storage, indexing strategies, and hybrid search patterns.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Installation & Setup](#installation--setup)\n3. [Schema Design](#schema-design)\n4. [Index Types](#index-types)\n5. [Query Patterns](#query-patterns)\n6. [Hybrid Search](#hybrid-search)\n7. [Performance Optimization](#performance-optimization)\n8. [Integration Patterns](#integration-patterns)\n\n## Overview\n\n### When to Use pgvector\n\n| Use Case | pgvector | Dedicated Vector DB |\n|----------|----------|---------------------|\n| \u003c 10M vectors | ✅ Excellent | Overkill |\n| Combined with SQL data | ✅ Natural fit | Requires sync |\n| Operational simplicity | ✅ One database | Multiple systems |\n| > 100M vectors | ⚠️ Consider limits | ✅ Better |\n| Real-time updates | ✅ Good | ✅ Good |\n| Filtering + similarity | ✅ Excellent | ⚠️ Varies |\n\n### Vector Search Concepts\n\n```mermaid\nflowchart LR\n subgraph INPUT[\"Input\"]\n TEXT[\"Text/Image\"]\n end\n\n subgraph EMBED[\"Embedding Model\"]\n OPENAI[\"OpenAI\"]\n COHERE[\"Cohere\"]\n LOCAL[\"Local Model\"]\n end\n\n subgraph STORAGE[\"PostgreSQL + pgvector\"]\n VECTOR[(\"Vectors\u003cbr/>[0.1, 0.3, ...]\")]\n INDEX[\"HNSW/IVFFlat\u003cbr/>Index\"]\n end\n\n subgraph OUTPUT[\"Results\"]\n SIMILAR[\"Similar Items\u003cbr/>+ Metadata\"]\n end\n\n TEXT --> EMBED\n EMBED --> VECTOR\n VECTOR --> INDEX\n INDEX --> SIMILAR\n\n style STORAGE fill:#c8e6c9\n```\n\n## Installation & Setup\n\n### Install pgvector Extension\n\n```sql\n-- Install extension (requires PostgreSQL 11+)\nCREATE EXTENSION IF NOT EXISTS vector;\n\n-- Verify installation\nSELECT * FROM pg_extension WHERE extname = 'vector';\n```\n\n### Vector Dimensions\n\n```sql\n-- Common embedding dimensions:\n-- OpenAI text-embedding-ada-002: 1536\n-- OpenAI text-embedding-3-small: 1536\n-- OpenAI text-embedding-3-large: 3072\n-- Cohere embed-english-v3.0: 1024\n-- sentence-transformers/all-MiniLM-L6-v2: 384\n-- CLIP: 512\n\n-- Maximum dimensions: 16,000 (pgvector 0.5+)\n```\n\n## Schema Design\n\n### Basic Vector Table\n\n```sql\nCREATE TABLE data.documents (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n title text NOT NULL,\n content text NOT NULL,\n embedding vector(1536), -- OpenAI ada-002 dimensions\n\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Comment on dimension choice\nCOMMENT ON COLUMN data.documents.embedding IS 'OpenAI text-embedding-ada-002 (1536 dimensions)';\n```\n\n### Multi-Vector Design (Different Embeddings)\n\n```sql\nCREATE TABLE data.products (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n name text NOT NULL,\n description text,\n image_url text,\n\n -- Different embeddings for different modalities\n name_embedding vector(1536), -- Text embedding\n desc_embedding vector(1536), -- Text embedding\n image_embedding vector(512), -- CLIP image embedding\n\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Index each embedding type separately\nCREATE INDEX products_name_embedding_idx ON data.products\n USING hnsw (name_embedding vector_cosine_ops);\n\nCREATE INDEX products_image_embedding_idx ON data.products\n USING hnsw (image_embedding vector_cosine_ops);\n```\n\n### Chunked Documents Pattern\n\n```sql\n-- Main document table\nCREATE TABLE data.documents (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n title text NOT NULL,\n source_url text,\n metadata jsonb NOT NULL DEFAULT '{}',\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\n-- Chunks with embeddings\nCREATE TABLE data.document_chunks (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n document_id uuid NOT NULL REFERENCES data.documents(id) ON DELETE CASCADE,\n chunk_index integer NOT NULL,\n content text NOT NULL,\n token_count integer NOT NULL,\n embedding vector(1536) NOT NULL,\n\n UNIQUE (document_id, chunk_index)\n);\n\nCREATE INDEX document_chunks_document_id_idx ON data.document_chunks(document_id);\nCREATE INDEX document_chunks_embedding_idx ON data.document_chunks\n USING hnsw (embedding vector_cosine_ops);\n```\n\n### Embedding with Metadata\n\n```sql\nCREATE TABLE data.embeddings (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n source_type text NOT NULL, -- 'document', 'product', 'user'\n source_id uuid NOT NULL,\n embedding vector(1536) NOT NULL,\n\n -- Metadata for filtering\n category text,\n language text,\n created_at timestamptz NOT NULL DEFAULT now(),\n\n UNIQUE (source_type, source_id)\n);\n\n-- Partial indexes for common filters\nCREATE INDEX embeddings_docs_idx ON data.embeddings\n USING hnsw (embedding vector_cosine_ops)\n WHERE source_type = 'document';\n\nCREATE INDEX embeddings_products_idx ON data.embeddings\n USING hnsw (embedding vector_cosine_ops)\n WHERE source_type = 'product';\n```\n\n## Index Types\n\n### HNSW (Hierarchical Navigable Small World)\n\n```sql\n-- HNSW: Best for most use cases\n-- Pros: Faster queries, better recall\n-- Cons: Slower build, more memory\n\nCREATE INDEX documents_embedding_hnsw_idx ON data.documents\n USING hnsw (embedding vector_cosine_ops);\n\n-- With tuning parameters\nCREATE INDEX documents_embedding_hnsw_idx ON data.documents\n USING hnsw (embedding vector_cosine_ops)\n WITH (m = 16, ef_construction = 64);\n\n-- Parameters:\n-- m: Max connections per node (default 16, higher = better recall, more memory)\n-- ef_construction: Build-time beam width (default 64, higher = slower build, better index)\n```\n\n### IVFFlat (Inverted File with Flat Compression)\n\n```sql\n-- IVFFlat: Faster to build, less memory\n-- Pros: Faster index creation, smaller size\n-- Cons: Requires training, lower recall\n\n-- Create index (requires data in table first)\nCREATE INDEX documents_embedding_ivf_idx ON data.documents\n USING ivfflat (embedding vector_cosine_ops)\n WITH (lists = 100);\n\n-- Rule of thumb for lists:\n-- lists = sqrt(row_count) for \u003c 1M rows\n-- lists = sqrt(row_count) to row_count/1000 for > 1M rows\n```\n\n### Distance Functions\n\n```sql\n-- Cosine distance (most common for text embeddings)\nCREATE INDEX idx_cosine ON data.documents\n USING hnsw (embedding vector_cosine_ops);\n\n-- L2 (Euclidean) distance\nCREATE INDEX idx_l2 ON data.documents\n USING hnsw (embedding vector_l2_ops);\n\n-- Inner product (for normalized vectors)\nCREATE INDEX idx_ip ON data.documents\n USING hnsw (embedding vector_ip_ops);\n\n-- Query operators:\n-- \u003c=> cosine distance\n-- \u003c-> L2 distance\n-- \u003c#> inner product (negative)\n```\n\n### Index Selection Guide\n\n| Scenario | Index Type | Distance | Parameters |\n|----------|------------|----------|------------|\n| General text search | HNSW | cosine | m=16, ef_construction=64 |\n| Large dataset (>10M) | IVFFlat | cosine | lists=sqrt(n) |\n| High recall needed | HNSW | cosine | m=32, ef_construction=128 |\n| Memory constrained | IVFFlat | cosine | lists=n/1000 |\n| Normalized vectors | HNSW | inner product | default |\n\n## Query Patterns\n\n### Basic Similarity Search\n\n```sql\n-- Find 10 most similar documents\nSELECT id, title, content, embedding \u003c=> '[0.1, 0.2, ...]'::vector AS distance\nFROM data.documents\nORDER BY embedding \u003c=> '[0.1, 0.2, ...]'::vector\nLIMIT 10;\n```\n\n### Similarity Search with Threshold\n\n```sql\n-- Find documents within distance threshold\nSELECT id, title, 1 - (embedding \u003c=> $1) AS similarity\nFROM data.documents\nWHERE embedding \u003c=> $1 \u003c 0.3 -- Distance threshold\nORDER BY embedding \u003c=> $1\nLIMIT 20;\n```\n\n### API Function for Similarity Search\n\n```sql\nCREATE FUNCTION api.search_similar_documents(\n in_query_embedding vector(1536),\n in_limit integer DEFAULT 10,\n in_min_similarity real DEFAULT 0.7\n)\nRETURNS TABLE (\n id uuid,\n title text,\n content text,\n similarity real\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT\n id,\n title,\n content,\n 1 - (embedding \u003c=> in_query_embedding) AS similarity\n FROM data.documents\n WHERE 1 - (embedding \u003c=> in_query_embedding) >= in_min_similarity\n ORDER BY embedding \u003c=> in_query_embedding\n LIMIT in_limit;\n$;\n```\n\n### K-Nearest Neighbors with Filter\n\n```sql\n-- Similarity search with category filter\nSELECT id, title, embedding \u003c=> $1 AS distance\nFROM data.documents\nWHERE category = 'technical'\n AND language = 'en'\nORDER BY embedding \u003c=> $1\nLIMIT 10;\n\n-- Note: Filter reduces index effectiveness\n-- Consider partial indexes for common filters\n```\n\n### Chunked Document Search with Context\n\n```sql\nCREATE FUNCTION api.search_documents(\n in_query_embedding vector(1536),\n in_limit integer DEFAULT 5\n)\nRETURNS TABLE (\n document_id uuid,\n document_title text,\n chunk_content text,\n similarity real,\n context_before text,\n context_after text\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n WITH ranked_chunks AS (\n SELECT\n c.document_id,\n c.chunk_index,\n c.content,\n 1 - (c.embedding \u003c=> in_query_embedding) AS similarity,\n ROW_NUMBER() OVER (PARTITION BY c.document_id ORDER BY c.embedding \u003c=> in_query_embedding) AS rn\n FROM data.document_chunks c\n ORDER BY c.embedding \u003c=> in_query_embedding\n LIMIT in_limit * 3 -- Get more for deduplication\n )\n SELECT\n d.id AS document_id,\n d.title AS document_title,\n rc.content AS chunk_content,\n rc.similarity,\n prev.content AS context_before,\n next.content AS context_after\n FROM ranked_chunks rc\n JOIN data.documents d ON d.id = rc.document_id\n LEFT JOIN data.document_chunks prev ON prev.document_id = rc.document_id\n AND prev.chunk_index = rc.chunk_index - 1\n LEFT JOIN data.document_chunks next ON next.document_id = rc.document_id\n AND next.chunk_index = rc.chunk_index + 1\n WHERE rc.rn = 1 -- Best chunk per document\n ORDER BY rc.similarity DESC\n LIMIT in_limit;\n$;\n```\n\n## Hybrid Search\n\n### Combine FTS with Vector Search\n\n```sql\n-- Table with both FTS and vector\nCREATE TABLE data.articles (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n title text NOT NULL,\n content text NOT NULL,\n embedding vector(1536),\n\n -- Full-text search vector\n search_vector tsvector GENERATED ALWAYS AS (\n setweight(to_tsvector('english', coalesce(title, '')), 'A') ||\n setweight(to_tsvector('english', coalesce(content, '')), 'B')\n ) STORED,\n\n created_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE INDEX articles_embedding_idx ON data.articles\n USING hnsw (embedding vector_cosine_ops);\nCREATE INDEX articles_search_idx ON data.articles\n USING gin (search_vector);\n```\n\n### Hybrid Search Function\n\n```sql\nCREATE FUNCTION api.hybrid_search(\n in_query_text text,\n in_query_embedding vector(1536),\n in_limit integer DEFAULT 10,\n in_vector_weight real DEFAULT 0.5 -- Balance between FTS and vector\n)\nRETURNS TABLE (\n id uuid,\n title text,\n content text,\n combined_score real,\n fts_score real,\n vector_score real\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n WITH fts_results AS (\n SELECT\n id,\n ts_rank(search_vector, websearch_to_tsquery('english', in_query_text)) AS score\n FROM data.articles\n WHERE search_vector @@ websearch_to_tsquery('english', in_query_text)\n ),\n vector_results AS (\n SELECT\n id,\n 1 - (embedding \u003c=> in_query_embedding) AS score\n FROM data.articles\n ORDER BY embedding \u003c=> in_query_embedding\n LIMIT 100 -- Pre-filter for performance\n ),\n combined AS (\n SELECT\n COALESCE(f.id, v.id) AS id,\n COALESCE(f.score, 0) AS fts_score,\n COALESCE(v.score, 0) AS vector_score\n FROM fts_results f\n FULL OUTER JOIN vector_results v ON f.id = v.id\n )\n SELECT\n a.id,\n a.title,\n a.content,\n (c.fts_score * (1 - in_vector_weight) + c.vector_score * in_vector_weight) AS combined_score,\n c.fts_score,\n c.vector_score\n FROM combined c\n JOIN data.articles a ON a.id = c.id\n ORDER BY combined_score DESC\n LIMIT in_limit;\n$;\n```\n\n### Reciprocal Rank Fusion (RRF)\n\n```sql\nCREATE FUNCTION api.rrf_search(\n in_query_text text,\n in_query_embedding vector(1536),\n in_limit integer DEFAULT 10,\n in_k integer DEFAULT 60 -- RRF constant\n)\nRETURNS TABLE (\n id uuid,\n title text,\n rrf_score real\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n WITH fts_ranked AS (\n SELECT\n id,\n ROW_NUMBER() OVER (ORDER BY ts_rank(search_vector, q) DESC) AS rank\n FROM data.articles, websearch_to_tsquery('english', in_query_text) AS q\n WHERE search_vector @@ q\n LIMIT 100\n ),\n vector_ranked AS (\n SELECT\n id,\n ROW_NUMBER() OVER (ORDER BY embedding \u003c=> in_query_embedding) AS rank\n FROM data.articles\n ORDER BY embedding \u003c=> in_query_embedding\n LIMIT 100\n ),\n rrf_scores AS (\n SELECT\n COALESCE(f.id, v.id) AS id,\n COALESCE(1.0 / (in_k + f.rank), 0) +\n COALESCE(1.0 / (in_k + v.rank), 0) AS rrf_score\n FROM fts_ranked f\n FULL OUTER JOIN vector_ranked v ON f.id = v.id\n )\n SELECT\n a.id,\n a.title,\n r.rrf_score\n FROM rrf_scores r\n JOIN data.articles a ON a.id = r.id\n ORDER BY r.rrf_score DESC\n LIMIT in_limit;\n$;\n```\n\n## Performance Optimization\n\n### Query Tuning\n\n```sql\n-- Set HNSW search parameters (per query)\nSET hnsw.ef_search = 100; -- Default 40, higher = better recall, slower\n\n-- Set IVFFlat search parameters\nSET ivfflat.probes = 10; -- Default 1, higher = better recall, slower\n\n-- For specific query\nBEGIN;\nSET LOCAL hnsw.ef_search = 200;\nSELECT * FROM api.search_similar_documents($1, 10);\nCOMMIT;\n```\n\n### Batch Vector Operations\n\n```sql\n-- Batch insert with COPY\nCOPY data.documents (id, title, content, embedding)\nFROM STDIN WITH (FORMAT csv);\n\n-- Batch similarity search\nCREATE FUNCTION api.batch_similarity_search(\n in_query_embeddings vector(1536)[],\n in_limit_per_query integer DEFAULT 5\n)\nRETURNS TABLE (\n query_index integer,\n document_id uuid,\n similarity real\n)\nLANGUAGE sql\nSTABLE\nAS $\n SELECT\n q.idx AS query_index,\n d.id AS document_id,\n 1 - (d.embedding \u003c=> q.embedding) AS similarity\n FROM unnest(in_query_embeddings) WITH ORDINALITY AS q(embedding, idx)\n CROSS JOIN LATERAL (\n SELECT id, embedding\n FROM data.documents\n ORDER BY embedding \u003c=> q.embedding\n LIMIT in_limit_per_query\n ) d\n ORDER BY q.idx, similarity DESC;\n$;\n```\n\n### Index Maintenance\n\n```sql\n-- Check index size\nSELECT\n indexrelname,\n pg_size_pretty(pg_relation_size(indexrelid)) AS size\nFROM pg_stat_user_indexes\nWHERE indexrelname LIKE '%embedding%';\n\n-- Rebuild index (if needed after many updates)\nREINDEX INDEX CONCURRENTLY documents_embedding_hnsw_idx;\n\n-- Vacuum to reclaim space\nVACUUM ANALYZE data.documents;\n```\n\n### Monitor Query Performance\n\n```sql\n-- Explain analyze vector query\nEXPLAIN (ANALYZE, BUFFERS)\nSELECT id, embedding \u003c=> '[...]'::vector AS distance\nFROM data.documents\nORDER BY embedding \u003c=> '[...]'::vector\nLIMIT 10;\n\n-- Should show: Index Scan using documents_embedding_hnsw_idx\n```\n\n## Integration Patterns\n\n### Store Embeddings from Application\n\n```sql\nCREATE PROCEDURE api.upsert_document_embedding(\n in_document_id uuid,\n in_title text,\n in_content text,\n in_embedding vector(1536)\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.documents (id, title, content, embedding)\n VALUES (in_document_id, in_title, in_content, in_embedding)\n ON CONFLICT (id) DO UPDATE\n SET title = EXCLUDED.title,\n content = EXCLUDED.content,\n embedding = EXCLUDED.embedding,\n updated_at = now();\nEND;\n$;\n```\n\n### Python Integration Example\n\n```python\nimport psycopg\nfrom openai import OpenAI\n\n# Generate embedding\nclient = OpenAI()\nresponse = client.embeddings.create(\n input=\"Your text here\",\n model=\"text-embedding-ada-002\"\n)\nembedding = response.data[0].embedding\n\n# Store in PostgreSQL\nwith psycopg.connect(\"postgresql://...\") as conn:\n conn.execute(\"\"\"\n INSERT INTO data.documents (title, content, embedding)\n VALUES (%s, %s, %s)\n \"\"\", (\"Title\", \"Content\", embedding))\n\n# Search\nwith psycopg.connect(\"postgresql://...\") as conn:\n results = conn.execute(\"\"\"\n SELECT id, title, 1 - (embedding \u003c=> %s) AS similarity\n FROM data.documents\n ORDER BY embedding \u003c=> %s\n LIMIT 10\n \"\"\", (embedding, embedding)).fetchall()\n```\n\n### RAG (Retrieval Augmented Generation) Pattern\n\n```sql\n-- Function for RAG context retrieval\nCREATE FUNCTION api.get_rag_context(\n in_query_embedding vector(1536),\n in_max_chunks integer DEFAULT 5,\n in_max_tokens integer DEFAULT 4000\n)\nRETURNS TABLE (\n context text,\n sources jsonb\n)\nLANGUAGE sql\nSTABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n WITH relevant_chunks AS (\n SELECT\n c.content,\n c.token_count,\n d.title AS source_title,\n d.source_url,\n 1 - (c.embedding \u003c=> in_query_embedding) AS similarity,\n SUM(c.token_count) OVER (ORDER BY c.embedding \u003c=> in_query_embedding) AS cumulative_tokens\n FROM data.document_chunks c\n JOIN data.documents d ON d.id = c.document_id\n ORDER BY c.embedding \u003c=> in_query_embedding\n LIMIT in_max_chunks * 2\n ),\n filtered_chunks AS (\n SELECT *\n FROM relevant_chunks\n WHERE cumulative_tokens \u003c= in_max_tokens\n LIMIT in_max_chunks\n )\n SELECT\n string_agg(content, E'\\n\\n---\\n\\n' ORDER BY similarity DESC) AS context,\n jsonb_agg(DISTINCT jsonb_build_object(\n 'title', source_title,\n 'url', source_url\n )) AS sources\n FROM filtered_chunks;\n$;\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":18484,"content_sha256":"78e3ac2388a98dfa67463801377454d81b1ba32fe4e999a2ced9eb27955edee1"},{"filename":"references/window-functions.md","content":"# Advanced Window Functions\n\nThis document covers PostgreSQL window functions including frame specifications, ranking, running calculations, and gap/island analysis.\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Window Function Syntax](#window-function-syntax)\n3. [Frame Specifications](#frame-specifications)\n4. [Ranking Functions](#ranking-functions)\n5. [Aggregate Window Functions](#aggregate-window-functions)\n6. [Value Functions](#value-functions)\n7. [Common Patterns](#common-patterns)\n8. [Performance Considerations](#performance-considerations)\n\n## Overview\n\n### When to Use Window Functions\n\n| Use Case | Window Function | Alternative |\n|----------|----------------|-------------|\n| Running total | `SUM() OVER` | Subquery (slower) |\n| Ranking | `ROW_NUMBER()`, `RANK()` | Subquery |\n| Moving average | `AVG() OVER` | Application code |\n| Comparing to previous | `LAG()`, `LEAD()` | Self-join |\n| First/last in group | `FIRST_VALUE()` | `DISTINCT ON` |\n| Percentiles | `PERCENT_RANK()` | Subquery |\n\n### Window vs Aggregate Functions\n\n```sql\n-- Aggregate: Collapses rows into one\nSELECT customer_id, SUM(total) AS total_sales\nFROM data.orders\nGROUP BY customer_id;\n\n-- Window: Keeps all rows, adds calculated column\nSELECT\n id,\n customer_id,\n total,\n SUM(total) OVER (PARTITION BY customer_id) AS customer_total\nFROM data.orders;\n```\n\n## Window Function Syntax\n\n### Basic Syntax\n\n```sql\nfunction_name(arguments) OVER (\n [PARTITION BY partition_expression, ...]\n [ORDER BY sort_expression [ASC | DESC] [NULLS {FIRST | LAST}], ...]\n [frame_clause]\n)\n```\n\n### PARTITION BY\n\n```sql\n-- Without PARTITION BY: All rows in one partition\nSELECT id, total, SUM(total) OVER () AS grand_total\nFROM data.orders;\n\n-- With PARTITION BY: Separate calculation per group\nSELECT\n id,\n customer_id,\n total,\n SUM(total) OVER (PARTITION BY customer_id) AS customer_total,\n SUM(total) OVER () AS grand_total\nFROM data.orders;\n```\n\n### ORDER BY\n\n```sql\n-- Running total (cumulative sum)\nSELECT\n id,\n created_at,\n total,\n SUM(total) OVER (ORDER BY created_at) AS running_total\nFROM data.orders;\n\n-- Running total per customer\nSELECT\n id,\n customer_id,\n created_at,\n total,\n SUM(total) OVER (PARTITION BY customer_id ORDER BY created_at) AS customer_running_total\nFROM data.orders;\n```\n\n### Named Windows (WINDOW Clause)\n\n```sql\n-- Define window once, reuse multiple times\nSELECT\n id,\n customer_id,\n total,\n SUM(total) OVER w AS running_total,\n AVG(total) OVER w AS running_avg,\n COUNT(*) OVER w AS running_count\nFROM data.orders\nWINDOW w AS (PARTITION BY customer_id ORDER BY created_at);\n```\n\n## Frame Specifications\n\n### Frame Types\n\n```sql\n-- ROWS: Physical rows\n-- RANGE: Logical range based on ORDER BY value\n-- GROUPS: Groups of peers (same ORDER BY value)\n\n-- Frame bounds:\n-- UNBOUNDED PRECEDING: First row of partition\n-- N PRECEDING: N rows/value before current\n-- CURRENT ROW: Current row\n-- N FOLLOWING: N rows/value after current\n-- UNBOUNDED FOLLOWING: Last row of partition\n```\n\n### ROWS Frame\n\n```sql\n-- Last 3 rows (physical)\nSELECT\n id,\n value,\n AVG(value) OVER (\n ORDER BY created_at\n ROWS BETWEEN 2 PRECEDING AND CURRENT ROW\n ) AS moving_avg_3\nFROM data.metrics;\n\n-- 5-row centered window\nSELECT\n id,\n value,\n AVG(value) OVER (\n ORDER BY created_at\n ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING\n ) AS centered_avg_5\nFROM data.metrics;\n```\n\n### RANGE Frame\n\n```sql\n-- Values within ±10 of current value\nSELECT\n id,\n score,\n COUNT(*) OVER (\n ORDER BY score\n RANGE BETWEEN 10 PRECEDING AND 10 FOLLOWING\n ) AS nearby_count\nFROM data.scores;\n\n-- Time-based: last 5 minutes\nSELECT\n id,\n recorded_at,\n value,\n AVG(value) OVER (\n ORDER BY recorded_at\n RANGE BETWEEN interval '5 minutes' PRECEDING AND CURRENT ROW\n ) AS avg_last_5min\nFROM data.events;\n```\n\n### GROUPS Frame\n\n```sql\n-- Include N groups of peers\nSELECT\n id,\n category,\n value,\n SUM(value) OVER (\n ORDER BY category\n GROUPS BETWEEN 1 PRECEDING AND 1 FOLLOWING\n ) AS sum_adjacent_categories\nFROM data.items;\n```\n\n### Default Frames\n\n```sql\n-- Without ORDER BY: entire partition\n-- RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING\n\n-- With ORDER BY: up to current row\n-- RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW\n\n-- Explicit full partition\nSUM(total) OVER (PARTITION BY customer_id ORDER BY created_at\n ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)\n```\n\n## Ranking Functions\n\n### ROW_NUMBER, RANK, DENSE_RANK\n\n```sql\nSELECT\n id,\n customer_id,\n total,\n -- ROW_NUMBER: Unique sequential numbers (1, 2, 3, 4, 5)\n ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY total DESC) AS row_num,\n\n -- RANK: Same rank for ties, gaps after (1, 2, 2, 4, 5)\n RANK() OVER (PARTITION BY customer_id ORDER BY total DESC) AS rank,\n\n -- DENSE_RANK: Same rank for ties, no gaps (1, 2, 2, 3, 4)\n DENSE_RANK() OVER (PARTITION BY customer_id ORDER BY total DESC) AS dense_rank\nFROM data.orders;\n```\n\n### NTILE\n\n```sql\n-- Divide into N buckets\nSELECT\n id,\n total,\n NTILE(4) OVER (ORDER BY total) AS quartile,\n NTILE(10) OVER (ORDER BY total) AS decile,\n NTILE(100) OVER (ORDER BY total) AS percentile_bucket\nFROM data.orders;\n```\n\n### PERCENT_RANK and CUME_DIST\n\n```sql\nSELECT\n id,\n total,\n -- PERCENT_RANK: Relative rank (0 to 1)\n -- (rank - 1) / (total rows - 1)\n PERCENT_RANK() OVER (ORDER BY total) AS percent_rank,\n\n -- CUME_DIST: Cumulative distribution\n -- rows with value \u003c= current / total rows\n CUME_DIST() OVER (ORDER BY total) AS cumulative_dist\nFROM data.orders;\n```\n\n### Top N per Group\n\n```sql\n-- Top 3 orders per customer\nSELECT *\nFROM (\n SELECT\n *,\n ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY total DESC) AS rn\n FROM data.orders\n) sub\nWHERE rn \u003c= 3;\n\n-- Using LATERAL (alternative)\nSELECT o.*\nFROM data.customers c\nCROSS JOIN LATERAL (\n SELECT *\n FROM data.orders\n WHERE customer_id = c.id\n ORDER BY total DESC\n LIMIT 3\n) o;\n```\n\n## Aggregate Window Functions\n\n### Running Totals\n\n```sql\n-- Running sum\nSELECT\n id,\n created_at,\n amount,\n SUM(amount) OVER (ORDER BY created_at) AS running_total\nFROM data.transactions;\n\n-- Running sum per account\nSELECT\n id,\n account_id,\n created_at,\n amount,\n SUM(amount) OVER (PARTITION BY account_id ORDER BY created_at) AS account_balance\nFROM data.transactions;\n```\n\n### Moving Averages\n\n```sql\n-- Simple moving average (last 7 rows)\nSELECT\n date,\n value,\n AVG(value) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS sma_7\nFROM data.daily_metrics;\n\n-- Exponential moving average (approximation)\n-- True EMA requires recursive CTE or custom aggregate\nSELECT\n date,\n value,\n AVG(value) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) *\n (2.0 / 8) + LAG(value) OVER (ORDER BY date) * (1 - 2.0 / 8) AS ema_approx\nFROM data.daily_metrics;\n```\n\n### Running Count and Statistics\n\n```sql\nSELECT\n id,\n created_at,\n value,\n COUNT(*) OVER (ORDER BY created_at) AS running_count,\n AVG(value) OVER (ORDER BY created_at) AS running_avg,\n MIN(value) OVER (ORDER BY created_at) AS running_min,\n MAX(value) OVER (ORDER BY created_at) AS running_max,\n STDDEV(value) OVER (ORDER BY created_at) AS running_stddev\nFROM data.measurements;\n```\n\n## Value Functions\n\n### LAG and LEAD\n\n```sql\nSELECT\n id,\n created_at,\n value,\n -- Previous value\n LAG(value) OVER (ORDER BY created_at) AS prev_value,\n\n -- Next value\n LEAD(value) OVER (ORDER BY created_at) AS next_value,\n\n -- N rows back (default if null)\n LAG(value, 3, 0) OVER (ORDER BY created_at) AS value_3_back,\n\n -- Change from previous\n value - LAG(value) OVER (ORDER BY created_at) AS change\nFROM data.metrics;\n```\n\n### FIRST_VALUE, LAST_VALUE, NTH_VALUE\n\n```sql\nSELECT\n id,\n customer_id,\n created_at,\n total,\n -- First order total for customer\n FIRST_VALUE(total) OVER (\n PARTITION BY customer_id ORDER BY created_at\n ) AS first_order_total,\n\n -- Last order total (need full frame!)\n LAST_VALUE(total) OVER (\n PARTITION BY customer_id ORDER BY created_at\n ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING\n ) AS last_order_total,\n\n -- Third order total\n NTH_VALUE(total, 3) OVER (\n PARTITION BY customer_id ORDER BY created_at\n ) AS third_order_total\nFROM data.orders;\n```\n\n## Common Patterns\n\n### Gap and Island Detection\n\n```sql\n-- Identify consecutive sequences (islands)\nWITH numbered AS (\n SELECT\n id,\n status,\n created_at,\n ROW_NUMBER() OVER (ORDER BY created_at) AS rn,\n ROW_NUMBER() OVER (PARTITION BY status ORDER BY created_at) AS status_rn\n FROM data.events\n),\nislands AS (\n SELECT\n id,\n status,\n created_at,\n rn - status_rn AS island_id\n FROM numbered\n)\nSELECT\n status,\n island_id,\n MIN(created_at) AS island_start,\n MAX(created_at) AS island_end,\n COUNT(*) AS island_size\nFROM islands\nGROUP BY status, island_id\nORDER BY island_start;\n```\n\n### Year-over-Year Comparison\n\n```sql\nSELECT\n date_trunc('month', created_at) AS month,\n SUM(total) AS revenue,\n LAG(SUM(total), 12) OVER (ORDER BY date_trunc('month', created_at)) AS revenue_last_year,\n ROUND(\n 100.0 * (SUM(total) - LAG(SUM(total), 12) OVER (ORDER BY date_trunc('month', created_at)))\n / NULLIF(LAG(SUM(total), 12) OVER (ORDER BY date_trunc('month', created_at)), 0),\n 2\n ) AS yoy_growth_pct\nFROM data.orders\nGROUP BY date_trunc('month', created_at)\nORDER BY month;\n```\n\n### Running Percentage of Total\n\n```sql\nSELECT\n id,\n category,\n total,\n ROUND(\n 100.0 * SUM(total) OVER (ORDER BY total DESC) /\n SUM(total) OVER (),\n 2\n ) AS cumulative_pct,\n ROUND(\n 100.0 * total / SUM(total) OVER (),\n 2\n ) AS pct_of_total\nFROM data.orders;\n```\n\n### Sessionization\n\n```sql\n-- Group events into sessions based on time gaps\nWITH event_gaps AS (\n SELECT\n id,\n user_id,\n created_at,\n created_at - LAG(created_at) OVER (PARTITION BY user_id ORDER BY created_at) AS gap\n FROM data.events\n),\nsession_markers AS (\n SELECT\n *,\n CASE\n WHEN gap IS NULL OR gap > interval '30 minutes'\n THEN 1\n ELSE 0\n END AS new_session\n FROM event_gaps\n)\nSELECT\n id,\n user_id,\n created_at,\n SUM(new_session) OVER (PARTITION BY user_id ORDER BY created_at) AS session_id\nFROM session_markers;\n```\n\n### Percentile Calculations\n\n```sql\nSELECT\n category,\n PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY value) AS median,\n PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY value) AS p25,\n PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY value) AS p75,\n PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY value) AS p95,\n PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY value) AS p99\nFROM data.metrics\nGROUP BY category;\n\n-- Per-row percentile (window mode)\nSELECT\n id,\n value,\n PERCENT_RANK() OVER (ORDER BY value) AS percentile,\n NTILE(100) OVER (ORDER BY value) AS percentile_bucket\nFROM data.metrics;\n```\n\n### Delta Encoding\n\n```sql\n-- Store differences instead of absolute values\nSELECT\n id,\n created_at,\n value,\n value - COALESCE(LAG(value) OVER (ORDER BY created_at), 0) AS delta\nFROM data.time_series;\n```\n\n## Performance Considerations\n\n### Index Support\n\n```sql\n-- Window functions benefit from indexes on:\n-- 1. PARTITION BY columns\n-- 2. ORDER BY columns\n-- 3. Combined (partition, order)\n\nCREATE INDEX orders_customer_created_idx\n ON data.orders (customer_id, created_at);\n\n-- Query uses index for both partitioning and ordering\nSELECT\n customer_id,\n created_at,\n total,\n SUM(total) OVER (PARTITION BY customer_id ORDER BY created_at) AS running_total\nFROM data.orders;\n```\n\n### Memory Usage\n\n```sql\n-- Large partitions consume more memory\n-- Check work_mem setting\nSHOW work_mem;\n\n-- Increase for complex window operations\nSET work_mem = '256MB';\n\n-- Or per-session\nSET LOCAL work_mem = '256MB';\n```\n\n### Avoiding Multiple Passes\n\n```sql\n-- ❌ Bad: Multiple window calculations with different partitions\nSELECT\n *,\n SUM(total) OVER (PARTITION BY customer_id) AS customer_total,\n SUM(total) OVER (PARTITION BY product_id) AS product_total\nFROM data.order_items;\n\n-- ✅ Better: Use named window or pre-compute\nWITH customer_totals AS (\n SELECT customer_id, SUM(total) AS total\n FROM data.order_items\n GROUP BY customer_id\n),\nproduct_totals AS (\n SELECT product_id, SUM(total) AS total\n FROM data.order_items\n GROUP BY product_id\n)\nSELECT\n oi.*,\n ct.total AS customer_total,\n pt.total AS product_total\nFROM data.order_items oi\nJOIN customer_totals ct ON oi.customer_id = ct.customer_id\nJOIN product_totals pt ON oi.product_id = pt.product_id;\n```\n\n### EXPLAIN ANALYZE\n\n```sql\n-- Check window function execution\nEXPLAIN (ANALYZE, BUFFERS)\nSELECT\n customer_id,\n created_at,\n total,\n SUM(total) OVER (PARTITION BY customer_id ORDER BY created_at) AS running_total\nFROM data.orders\nWHERE created_at >= '2024-01-01';\n\n-- Look for: WindowAgg, Sort, Index Scan\n```\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":13369,"content_sha256":"0c025c4f73360c37ae2a7869940640f1907239d1928b3e8bb131e805bae14eff"},{"filename":"tests/config/test_config.env","content":"# ============================================================================\n# TEST CONFIGURATION\n# ============================================================================\n# Environment variables for the test suite.\n# Copy this file and customize for your environment.\n#\n# Usage:\n# source test_config.env\n# ./scripts/run_all_tests.sh\n# ============================================================================\n\n# Database connection\nexport PGHOST=\"${PGHOST:-localhost}\"\nexport PGPORT=\"${PGPORT:-5432}\"\nexport PGUSER=\"${PGUSER:-postgres}\"\nexport PGDATABASE=\"${PGDATABASE:-postgres}\"\n# export PGPASSWORD=\"\" # Uncomment and set if needed\n\n# Test options\nexport TEST_VERBOSE=\"${TEST_VERBOSE:-false}\"\nexport TEST_SKIP_CLEANUP=\"${TEST_SKIP_CLEANUP:-false}\"\n\n# CI/CD options\nexport CI=\"${CI:-false}\"\n","content_type":"text/plain; charset=utf-8","language":"bash","size":808,"content_sha256":"3bf62738079f00f0027541ffd8ccfe20044196ed145d00ecabe3e113fb3c5ae4"},{"filename":"tests/README.md","content":"# PostgreSQL Best Practices - Test Suite\n\nComprehensive test suite for validating the PostgreSQL Best Practices Claude Skill.\n\n## Prerequisites\n\n- **PostgreSQL 18+** (recommended) or PostgreSQL 14+ (minimum)\n- `psql` command-line client\n- Database with admin/superuser privileges\n- Bash shell (for running scripts)\n\n## Quick Start\n\n```bash\n# 1. Navigate to tests directory\ncd tests\n\n# 2. Set connection (optional - uses defaults if not set)\nexport PGHOST=localhost\nexport PGPORT=5432\nexport PGUSER=postgres\nexport PGDATABASE=testdb\n\n# 3. Run all tests\n./scripts/run_all_tests.sh\n```\n\n## Directory Structure\n\n```\ntests/\n├── README.md # This file\n├── setup/\n│ ├── 00_check_prerequisites.sql # Verify PG version and extensions\n│ └── 01_install_test_framework.sql # Install test schema and functions\n├── teardown/\n│ └── 01_cleanup.sql # Clean up test data\n├── framework/\n│ ├── assertions.sql # Core assertion functions\n│ ├── test_runner.sql # Test discovery and execution\n│ └── test_helpers.sql # Utilities and data factories\n├── modules/\n│ ├── 01_migration_system/ # Migration system tests (~40 tests)\n│ ├── 02_schema_architecture/ # Schema pattern tests (~10 tests)\n│ ├── 03_plpgsql_patterns/ # PL/pgSQL convention tests (~15 tests)\n│ ├── 04_data_types/ # Data type tests (~12 tests)\n│ └── 05_anti_patterns/ # Anti-pattern detection (~8 tests)\n├── integration/\n│ ├── 010_full_workflow_test.sql # End-to-end workflow\n│ └── 020_concurrent_access_test.sql # Locking behavior\n├── scripts/\n│ ├── run_all_tests.sh # Run complete suite\n│ ├── run_module.sh # Run specific module\n│ └── ci_runner.sh # CI/CD optimized runner\n└── config/\n └── test_config.env # Environment configuration\n```\n\n## Running Tests\n\n### Run All Tests\n\n```bash\n./scripts/run_all_tests.sh\n```\n\nOptions:\n- `-d, --database \u003cname>`: Database name\n- `-h, --host \u003chost>`: Database host\n- `-p, --port \u003cport>`: Database port\n- `-U, --user \u003cuser>`: Database user\n- `-v, --verbose`: Show detailed output\n- `--skip-setup`: Skip framework installation\n- `--skip-cleanup`: Keep test data after run\n\n### Run Specific Module\n\n```bash\n./scripts/run_module.sh 01_migration_system\n./scripts/run_module.sh 02_schema_architecture\n./scripts/run_module.sh integration\n```\n\n### Run in CI/CD\n\n```bash\n./scripts/ci_runner.sh\n```\n\nExit codes:\n- `0`: All tests passed\n- `1`: One or more tests failed\n- `2`: Setup/connection error\n\n### Run Individual Test File\n\n```bash\npsql -d testdb -f modules/01_migration_system/020_locking_test.sql\n```\n\n## Test Framework API\n\n### Assertions\n\n| Function | Description |\n|----------|-------------|\n| `test.ok(condition, description)` | Pass if condition is true |\n| `test.is(got, expected, description)` | Pass if values match |\n| `test.isnt(got, unexpected, description)` | Pass if values differ |\n| `test.is_null(value, description)` | Pass if value is NULL |\n| `test.is_not_null(value, description)` | Pass if value is not NULL |\n| `test.throws_ok(sql, errcode, description)` | Pass if SQL throws expected error |\n| `test.throws_like(sql, pattern, description)` | Pass if error matches pattern |\n| `test.lives_ok(sql, description)` | Pass if SQL executes without error |\n| `test.has_schema(name, description)` | Pass if schema exists |\n| `test.has_table(schema, table, description)` | Pass if table exists |\n| `test.has_function(schema, func, description)` | Pass if function exists |\n| `test.has_procedure(schema, proc, description)` | Pass if procedure exists |\n| `test.has_column(schema, table, col, description)` | Pass if column exists |\n| `test.has_index(schema, table, idx, description)` | Pass if index exists |\n| `test.row_count_is(query, count, description)` | Pass if query returns expected count |\n| `test.is_empty(query, description)` | Pass if query returns no rows |\n| `test.matches(value, pattern, description)` | Pass if value matches regex |\n\n### Test Execution\n\n| Function | Description |\n|----------|-------------|\n| `test.set_context(name)` | Set current test name |\n| `test.run_test(func_name)` | Run a single test function |\n| `test.run_all(schema, pattern)` | Run all tests matching pattern |\n| `test.run_module(module_name)` | Run tests for a module |\n| `test.print_summary()` | Print formatted results |\n\n### Helpers\n\n| Function | Description |\n|----------|-------------|\n| `test.unique_id()` | Generate unique test identifier |\n| `test.test_email(suffix)` | Generate test email address |\n| `test.begin_test(name)` | Create savepoint for isolation |\n| `test.rollback_test(name)` | Rollback to savepoint |\n| `test.exec_count(sql)` | Execute and return row count |\n| `test.measure_time(sql)` | Measure execution time (ms) |\n| `test.is_secure_function(schema, func)` | Check SECURITY DEFINER + search_path |\n\n## Writing New Tests\n\n### Test Function Convention\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_\u003cmodule>_\u003cnumber>_\u003cdescription>()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nBEGIN\n PERFORM test.set_context('test_\u003cmodule>_\u003cnumber>_\u003cdescription>');\n\n -- Your assertions here\n PERFORM test.ok(true, 'description');\nEND;\n$;\n```\n\n### Example Test\n\n```sql\nCREATE OR REPLACE FUNCTION test.test_example_010_basic()\nRETURNS void\nLANGUAGE plpgsql\nAS $\nDECLARE\n l_result integer;\nBEGIN\n PERFORM test.set_context('test_example_010_basic');\n\n -- Test basic assertion\n l_result := 1 + 1;\n PERFORM test.is(l_result, 2, '1 + 1 should equal 2');\n\n -- Test SQL execution\n PERFORM test.lives_ok('SELECT 1', 'SELECT should succeed');\n\n -- Test error handling\n PERFORM test.throws_ok(\n 'SELECT 1/0',\n '22012', -- division_by_zero\n 'Division by zero should throw'\n );\nEND;\n$;\n```\n\n### Test Naming Convention\n\n- `test_\u003cmodule>_\u003cNNN>_\u003cdescription>`\n- Module prefixes match directory numbers (e.g., `migration_01`, `schema_02`)\n- Numbers should be sequential within each test file (010, 011, 020, etc.)\n\n## Test Modules\n\n### 01_migration_system (~40 tests)\n\nTests for the native PL/pgSQL migration system:\n- Installation and schema structure\n- Lock acquire/release/timeout\n- Versioned migration execution\n- Repeatable migration change detection\n- Checksum calculation and validation\n- Rollback functionality\n- Batch execution\n- Status and info queries\n\n### 02_schema_architecture (~10 tests)\n\nTests for three-schema separation pattern:\n- data/private/api schema existence\n- SECURITY DEFINER with SET search_path\n- Role-based access control\n\n### 03_plpgsql_patterns (~15 tests)\n\nTests for PL/pgSQL conventions:\n- Trivadis naming conventions (l_, in_, io_, co_, r_, c_, t_)\n- Table API pattern (functions for reads, procedures for writes)\n- Trigger patterns (updated_at, audit logging)\n- Error handling with SQLSTATE codes\n\n### 04_data_types (~12 tests)\n\nTests for data type recommendations:\n- UUIDv7 generation and properties\n- timestamptz vs timestamp handling\n- numeric precision for financial data\n- JSONB storage, querying, and indexing\n\n### 05_anti_patterns (~8 tests)\n\nTests demonstrating correct patterns vs anti-patterns:\n- NOT EXISTS vs NOT IN with NULLs\n- >= AND \u003c vs BETWEEN for date ranges\n- Missing FK index detection\n- SECURITY DEFINER without search_path\n\n## Troubleshooting\n\n### Connection Issues\n\n```bash\n# Test connection\npsql -h localhost -p 5432 -U postgres -d testdb -c \"SELECT 1\"\n\n# Check PostgreSQL version\npsql -c \"SELECT version()\"\n```\n\n### Permission Issues\n\nThe test framework requires privileges to:\n- Create schemas (test, data, private, api)\n- Create functions and procedures\n- Create and drop tables\n- Execute advisory locks\n\n### Cleanup Stuck Locks\n\n```sql\n-- Release any held migration locks\nSELECT app_migration.release_lock();\n\n-- Check lock status\nSELECT * FROM app_migration.get_lock_holder();\n```\n\n### Reset Test Framework\n\n```sql\n-- Clear all test results\nCALL test.clear_results();\n\n-- Reinstall framework\n\\i setup/01_install_test_framework.sql\n```\n\n## Output Format\n\nTests output TAP (Test Anything Protocol) format:\n\n```\nok 1 - schema exists\nok 2 - table created\nnot ok 3 - expected 5 rows, got 3\n# got: 3\n# expected: 5\n```\n\n## License\n\nPart of the PostgreSQL Best Practices Claude Skill repository.\n","content_type":"text/markdown; charset=utf-8","language":"markdown","size":8533,"content_sha256":"4252564e6bb087ac52a967268122bdab22cbfed04ab418a272b2911998580a3d"},{"filename":"tests/scripts/ci_runner.sh","content":"#!/bin/bash\n# ============================================================================\n# CI/CD TEST RUNNER\n# ============================================================================\n# Optimized test runner for CI/CD pipelines.\n# Returns proper exit codes and minimal output.\n#\n# Exit codes:\n# 0 - All tests passed\n# 1 - One or more tests failed\n# 2 - Setup/connection error\n#\n# Usage: ./ci_runner.sh [OPTIONS]\n#\n# Environment variables:\n# PGHOST, PGPORT, PGUSER, PGDATABASE, PGPASSWORD\n# ============================================================================\n\nset -e\n\n# Script directory\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nTESTS_DIR=\"$(dirname \"$SCRIPT_DIR\")\"\n\n# Connection parameters from environment\nDB_NAME=\"${PGDATABASE:-postgres}\"\nDB_HOST=\"${PGHOST:-localhost}\"\nDB_PORT=\"${PGPORT:-5432}\"\nDB_USER=\"${PGUSER:-postgres}\"\n\nPSQL=\"psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -v ON_ERROR_STOP=1\"\n\n# Logging\nlog() {\n echo \"[$(date '+%Y-%m-%d %H:%M:%S')] $1\"\n}\n\nerror() {\n echo \"[$(date '+%Y-%m-%d %H:%M:%S')] ERROR: $1\" >&2\n}\n\n# ============================================================================\n# PRE-FLIGHT CHECKS\n# ============================================================================\n\nlog \"Starting CI test run...\"\nlog \"Database: $DB_NAME @ $DB_HOST:$DB_PORT (user: $DB_USER)\"\n\n# Test connection\nif ! $PSQL -c \"SELECT 1\" &>/dev/null; then\n error \"Cannot connect to database\"\n exit 2\nfi\n\n# Check PostgreSQL version\nPG_VERSION=$($PSQL -t -A -c \"SHOW server_version_num\")\nlog \"PostgreSQL version: $PG_VERSION\"\n\nif [ \"$PG_VERSION\" -lt 140000 ]; then\n error \"PostgreSQL 14+ required (found: $PG_VERSION)\"\n exit 2\nfi\n\n# ============================================================================\n# SETUP\n# ============================================================================\n\nlog \"Installing test framework...\"\n\ncd \"$TESTS_DIR/setup\"\n$PSQL -f \"01_install_test_framework.sql\" -q 2>/dev/null\n\ncd \"$TESTS_DIR/../scripts\"\n$PSQL -f \"001_install_migration_system.sql\" -q 2>/dev/null\n$PSQL -f \"002_migration_runner_helpers.sql\" -q 2>/dev/null\n\n# Clear old test results\n$PSQL -c \"TRUNCATE test.results\" -q 2>/dev/null || true\n$PSQL -c \"TRUNCATE test.runs CASCADE\" -q 2>/dev/null || true\n\n# ============================================================================\n# RUN TESTS\n# ============================================================================\n\nlog \"Running test modules...\"\n\nrun_tests() {\n local dir=$1\n local name=$2\n\n if [ -d \"$dir\" ]; then\n log \" Module: $name\"\n for f in \"$dir\"/*.sql; do\n [ -f \"$f\" ] && $PSQL -f \"$f\" -q 2>&1 | grep -E \"^(not ok)\" || true\n done\n fi\n}\n\n# Run all modules\nrun_tests \"$TESTS_DIR/modules/01_migration_system\" \"migration_system\"\nrun_tests \"$TESTS_DIR/modules/02_schema_architecture\" \"schema_architecture\"\nrun_tests \"$TESTS_DIR/modules/03_plpgsql_patterns\" \"plpgsql_patterns\"\nrun_tests \"$TESTS_DIR/modules/04_data_types\" \"data_types\"\nrun_tests \"$TESTS_DIR/modules/05_anti_patterns\" \"anti_patterns\"\nrun_tests \"$TESTS_DIR/integration\" \"integration\"\n\n# ============================================================================\n# RESULTS\n# ============================================================================\n\nlog \"Collecting results...\"\n\n# Get results\nRESULTS=$($PSQL -t -A -c \"\n SELECT\n count(*) FILTER (WHERE passed),\n count(*) FILTER (WHERE NOT passed),\n count(*)\n FROM test.results\n\")\n\nPASSED=$(echo \"$RESULTS\" | cut -d'|' -f1)\nFAILED=$(echo \"$RESULTS\" | cut -d'|' -f2)\nTOTAL=$(echo \"$RESULTS\" | cut -d'|' -f3)\n\nlog \"Results: $PASSED passed, $FAILED failed ($TOTAL total)\"\n\n# Output failed tests for CI logs\nif [ \"$FAILED\" -gt 0 ]; then\n echo \"\"\n echo \"FAILED TESTS:\"\n echo \"=============\"\n $PSQL -t -A -c \"\n SELECT test_name || ': ' || description || ' (got: ' || COALESCE(got, 'NULL') || ', expected: ' || COALESCE(expected, 'NULL') || ')'\n FROM test.results\n WHERE NOT passed\n ORDER BY executed_at\n LIMIT 50\n \"\nfi\n\n# ============================================================================\n# CLEANUP\n# ============================================================================\n\nlog \"Cleaning up...\"\ncd \"$TESTS_DIR/teardown\"\n$PSQL -f \"01_cleanup.sql\" -q 2>/dev/null || true\n\n# ============================================================================\n# EXIT\n# ============================================================================\n\nif [ \"$FAILED\" -gt 0 ]; then\n log \"TEST RUN FAILED\"\n exit 1\nelse\n log \"TEST RUN PASSED\"\n exit 0\nfi\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":4625,"content_sha256":"e2a36a6565bdceb968d64019a162ffe8a4aea39f534631cb27c9be77b0b127b2"},{"filename":"tests/scripts/run_all_tests.sh","content":"#!/bin/bash\n# ============================================================================\n# RUN ALL TESTS\n# ============================================================================\n# Executes the complete test suite for PostgreSQL Best Practices Skill.\n#\n# Usage: ./run_all_tests.sh [OPTIONS]\n#\n# Options:\n# -d, --database Database name (default: from PGDATABASE or postgres)\n# -h, --host Database host (default: from PGHOST or localhost)\n# -p, --port Database port (default: from PGPORT or 5432)\n# -U, --user Database user (default: from PGUSER or current user)\n# -v, --verbose Show verbose output\n# --skip-setup Skip framework installation\n# --skip-cleanup Skip cleanup after tests\n# --help Show this help message\n# ============================================================================\n\nset -e\n\n# Default values (use environment variables if set)\nDB_NAME=\"${PGDATABASE:-postgres}\"\nDB_HOST=\"${PGHOST:-localhost}\"\nDB_PORT=\"${PGPORT:-5432}\"\nDB_USER=\"${PGUSER:-$USER}\"\nVERBOSE=false\nSKIP_SETUP=false\nSKIP_CLEANUP=false\n\n# Script directory\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nTESTS_DIR=\"$(dirname \"$SCRIPT_DIR\")\"\n\n# Colors for output\nRED='\\033[0;31m'\nGREEN='\\033[0;32m'\nYELLOW='\\033[1;33m'\nBLUE='\\033[0;34m'\nNC='\\033[0m' # No Color\n\n# Parse arguments\nwhile [[ $# -gt 0 ]]; do\n case $1 in\n -d|--database)\n DB_NAME=\"$2\"\n shift 2\n ;;\n -h|--host)\n DB_HOST=\"$2\"\n shift 2\n ;;\n -p|--port)\n DB_PORT=\"$2\"\n shift 2\n ;;\n -U|--user)\n DB_USER=\"$2\"\n shift 2\n ;;\n -v|--verbose)\n VERBOSE=true\n shift\n ;;\n --skip-setup)\n SKIP_SETUP=true\n shift\n ;;\n --skip-cleanup)\n SKIP_CLEANUP=true\n shift\n ;;\n --help)\n head -25 \"$0\" | tail -20\n exit 0\n ;;\n *)\n echo \"Unknown option: $1\"\n exit 1\n ;;\n esac\ndone\n\n# psql command with connection parameters\nPSQL=\"psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME\"\n\n# Helper functions\nlog_info() {\n echo -e \"${BLUE}[INFO]${NC} $1\"\n}\n\nlog_success() {\n echo -e \"${GREEN}[PASS]${NC} $1\"\n}\n\nlog_error() {\n echo -e \"${RED}[FAIL]${NC} $1\"\n}\n\nlog_warning() {\n echo -e \"${YELLOW}[WARN]${NC} $1\"\n}\n\nrun_sql_file() {\n local file=$1\n local name=$(basename \"$file\" .sql)\n\n if $VERBOSE; then\n $PSQL -f \"$file\" 2>&1\n else\n $PSQL -f \"$file\" -q 2>&1 | grep -E \"^(ok|not ok|#|NOTICE:.*test)\" || true\n fi\n}\n\n# ============================================================================\n# MAIN EXECUTION\n# ============================================================================\n\necho \"\"\necho \"============================================================\"\necho \"PostgreSQL Best Practices - Test Suite\"\necho \"============================================================\"\necho \"\"\nlog_info \"Database: $DB_NAME @ $DB_HOST:$DB_PORT\"\nlog_info \"User: $DB_USER\"\necho \"\"\n\n# Check prerequisites\nlog_info \"Checking prerequisites...\"\nif ! command -v psql &> /dev/null; then\n log_error \"psql command not found. Please install PostgreSQL client.\"\n exit 1\nfi\n\n# Test connection\nif ! $PSQL -c \"SELECT 1\" &> /dev/null; then\n log_error \"Cannot connect to database. Check connection parameters.\"\n exit 1\nfi\n\nlog_success \"Database connection OK\"\n\n# Install test framework\nif ! $SKIP_SETUP; then\n echo \"\"\n log_info \"Installing test framework...\"\n cd \"$TESTS_DIR/setup\"\n\n # Check prerequisites\n run_sql_file \"00_check_prerequisites.sql\"\n\n # Install framework\n run_sql_file \"01_install_test_framework.sql\"\n\n log_success \"Test framework installed\"\nfi\n\n# Install migration system if needed\nlog_info \"Ensuring migration system is installed...\"\ncd \"$TESTS_DIR/../scripts\"\n$PSQL -f \"001_install_migration_system.sql\" -q 2>/dev/null || true\n$PSQL -f \"002_migration_runner_helpers.sql\" -q 2>/dev/null || true\nlog_success \"Migration system ready\"\n\n# Run tests by module\necho \"\"\necho \"============================================================\"\necho \"Running Test Modules\"\necho \"============================================================\"\n\nTOTAL_PASSED=0\nTOTAL_FAILED=0\nMODULES_RUN=0\n\nrun_module() {\n local module_dir=$1\n local module_name=$(basename \"$module_dir\")\n\n echo \"\"\n log_info \"Module: $module_name\"\n echo \"------------------------------------------------------------\"\n\n # Run all test files in the module directory in order\n for test_file in \"$module_dir\"/*.sql; do\n if [ -f \"$test_file\" ]; then\n local test_name=$(basename \"$test_file\" .sql)\n log_info \" Running: $test_name\"\n run_sql_file \"$test_file\"\n fi\n done\n\n MODULES_RUN=$((MODULES_RUN + 1))\n}\n\n# Run all modules in order (sorted alphabetically)\nfor module_dir in \"$TESTS_DIR/modules\"/*; do\n if [ -d \"$module_dir\" ]; then\n run_module \"$module_dir\"\n fi\ndone\n\n# Integration Tests\nif [ -d \"$TESTS_DIR/integration\" ]; then\n echo \"\"\n log_info \"Integration Tests\"\n echo \"------------------------------------------------------------\"\n for test_file in \"$TESTS_DIR/integration\"/*.sql; do\n if [ -f \"$test_file\" ]; then\n test_name=$(basename \"$test_file\" .sql)\n log_info \" Running: $test_name\"\n run_sql_file \"$test_file\"\n fi\n done\nfi\n\n# Get final results\necho \"\"\necho \"============================================================\"\necho \"Test Results Summary\"\necho \"============================================================\"\n\n# Query test results from database\nRESULTS=$($PSQL -t -A -c \"\n SELECT\n count(*) FILTER (WHERE passed) as passed,\n count(*) FILTER (WHERE NOT passed) as failed,\n count(*) as total\n FROM test.results\n WHERE executed_at > now() - interval '1 hour'\n\" 2>/dev/null || echo \"0|0|0\")\n\nPASSED=$(echo \"$RESULTS\" | cut -d'|' -f1)\nFAILED=$(echo \"$RESULTS\" | cut -d'|' -f2)\nTOTAL=$(echo \"$RESULTS\" | cut -d'|' -f3)\n\necho \"\"\necho \"Modules run: $MODULES_RUN\"\necho \"Total assertions: $TOTAL\"\necho -e \"Passed: ${GREEN}$PASSED${NC}\"\necho -e \"Failed: ${RED}$FAILED${NC}\"\n\nif [ \"$TOTAL\" -gt 0 ]; then\n PCT=$(echo \"scale=1; $PASSED * 100 / $TOTAL\" | bc)\n echo \"Success rate: $PCT%\"\nfi\n\n# Show failed tests if any\nif [ \"$FAILED\" -gt 0 ]; then\n echo \"\"\n log_error \"Failed assertions:\"\n $PSQL -c \"\n SELECT test_name, description, got, expected\n FROM test.results\n WHERE NOT passed\n AND executed_at > now() - interval '1 hour'\n ORDER BY executed_at\n LIMIT 20\n \" 2>/dev/null || true\nfi\n\n# Cleanup\nif ! $SKIP_CLEANUP; then\n echo \"\"\n log_info \"Cleaning up test data...\"\n cd \"$TESTS_DIR/teardown\"\n $PSQL -f \"01_cleanup.sql\" -q 2>/dev/null || true\n log_success \"Cleanup complete\"\nfi\n\necho \"\"\necho \"============================================================\"\nif [ \"$FAILED\" -gt 0 ]; then\n log_error \"TEST SUITE FAILED\"\n exit 1\nelse\n log_success \"TEST SUITE PASSED\"\n exit 0\nfi\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":7214,"content_sha256":"b25da7dfdc982a317c3cd398e730244e8cd4e8171af48cdbd8c47df2b9607528"},{"filename":"tests/scripts/run_module.sh","content":"#!/bin/bash\n# ============================================================================\n# RUN SPECIFIC MODULE\n# ============================================================================\n# Runs tests for a specific module only.\n#\n# Usage: ./run_module.sh \u003cmodule_name> [OPTIONS]\n#\n# Modules:\n# 01_migration_system\n# 02_schema_architecture\n# 03_plpgsql_patterns\n# 04_data_types\n# 05_anti_patterns\n# integration\n#\n# Options:\n# -d, --database Database name\n# -v, --verbose Show verbose output\n# --help Show this help message\n# ============================================================================\n\nset -e\n\n# Check for module argument\nif [ -z \"$1\" ] || [ \"$1\" = \"--help\" ]; then\n head -20 \"$0\" | tail -18\n exit 0\nfi\n\nMODULE_NAME=$1\nshift\n\n# Default values\nDB_NAME=\"${PGDATABASE:-postgres}\"\nDB_HOST=\"${PGHOST:-localhost}\"\nDB_PORT=\"${PGPORT:-5432}\"\nDB_USER=\"${PGUSER:-$USER}\"\nVERBOSE=false\n\n# Script directory\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nTESTS_DIR=\"$(dirname \"$SCRIPT_DIR\")\"\n\n# Colors\nRED='\\033[0;31m'\nGREEN='\\033[0;32m'\nBLUE='\\033[0;34m'\nNC='\\033[0m'\n\n# Parse arguments\nwhile [[ $# -gt 0 ]]; do\n case $1 in\n -d|--database)\n DB_NAME=\"$2\"\n shift 2\n ;;\n -v|--verbose)\n VERBOSE=true\n shift\n ;;\n *)\n shift\n ;;\n esac\ndone\n\n# psql command\nPSQL=\"psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME\"\n\n# Determine module directory\nif [ \"$MODULE_NAME\" = \"integration\" ]; then\n MODULE_DIR=\"$TESTS_DIR/integration\"\nelse\n MODULE_DIR=\"$TESTS_DIR/modules/$MODULE_NAME\"\nfi\n\nif [ ! -d \"$MODULE_DIR\" ]; then\n echo -e \"${RED}[ERROR]${NC} Module not found: $MODULE_NAME\"\n echo \"\"\n echo \"Available modules:\"\n ls -1 \"$TESTS_DIR/modules\" 2>/dev/null || true\n echo \"integration\"\n exit 1\nfi\n\necho \"\"\necho \"============================================================\"\necho \"Running Module: $MODULE_NAME\"\necho \"============================================================\"\necho \"\"\n\n# Ensure test framework is installed\n$PSQL -c \"SELECT 1 FROM test.results LIMIT 0\" &>/dev/null || {\n echo -e \"${BLUE}[INFO]${NC} Installing test framework...\"\n cd \"$TESTS_DIR/setup\"\n $PSQL -f \"01_install_test_framework.sql\" -q\n}\n\n# Ensure migration system is installed\n$PSQL -c \"SELECT 1 FROM app_migration.changelog LIMIT 0\" &>/dev/null || {\n echo -e \"${BLUE}[INFO]${NC} Installing migration system...\"\n cd \"$TESTS_DIR/../scripts\"\n $PSQL -f \"001_install_migration_system.sql\" -q\n $PSQL -f \"002_migration_runner_helpers.sql\" -q\n}\n\n# Clear previous results for this run\n$PSQL -c \"DELETE FROM test.results WHERE executed_at \u003c now() - interval '1 minute'\" -q 2>/dev/null || true\n\n# Run test files\nfor test_file in \"$MODULE_DIR\"/*.sql; do\n if [ -f \"$test_file\" ]; then\n test_name=$(basename \"$test_file\" .sql)\n echo -e \"${BLUE}[INFO]${NC} Running: $test_name\"\n\n if $VERBOSE; then\n $PSQL -f \"$test_file\" 2>&1\n else\n $PSQL -f \"$test_file\" -q 2>&1 | grep -E \"^(ok|not ok|#)\" || true\n fi\n\n echo \"\"\n fi\ndone\n\n# Show results\necho \"============================================================\"\necho \"Results\"\necho \"============================================================\"\n\nRESULTS=$($PSQL -t -A -c \"\n SELECT\n count(*) FILTER (WHERE passed) as passed,\n count(*) FILTER (WHERE NOT passed) as failed,\n count(*) as total\n FROM test.results\n WHERE executed_at > now() - interval '5 minutes'\n\" 2>/dev/null || echo \"0|0|0\")\n\nPASSED=$(echo \"$RESULTS\" | cut -d'|' -f1)\nFAILED=$(echo \"$RESULTS\" | cut -d'|' -f2)\nTOTAL=$(echo \"$RESULTS\" | cut -d'|' -f3)\n\necho \"\"\necho \"Total: $TOTAL\"\necho -e \"Passed: ${GREEN}$PASSED${NC}\"\necho -e \"Failed: ${RED}$FAILED${NC}\"\n\nif [ \"$FAILED\" -gt 0 ]; then\n echo \"\"\n echo -e \"${RED}Failed assertions:${NC}\"\n $PSQL -c \"\n SELECT test_name, description, got, expected\n FROM test.results\n WHERE NOT passed\n AND executed_at > now() - interval '5 minutes'\n ORDER BY executed_at\n \" 2>/dev/null || true\n exit 1\nfi\n\nexit 0\n","content_type":"application/x-sh; charset=utf-8","language":"bash","size":4178,"content_sha256":"9df8d71a8d49461cec1b627b6c39a535c386827d4a93da7edd6298185a252990"}],"content_json":{"type":"doc","content":[{"type":"heading","attrs":{"level":1},"content":[{"text":"PostgreSQL Advanced Best Practices (PostgreSQL 18+)","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Architecture at a Glance","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":" ┌─── PostgreSQL Database ──────────────────────────────┐\n │ │\n │ ┌──────────────────┐ ┌───────────────────────┐ │\n │ │ api schema │ │ private schema │ │\n ┌─────────────┐ │ │──────────────────│ │───────────────────────│ │\n │ Application │─EXECUTE─▶│ get_customer() │───▶│ set_updated_at() │ │\n └─────────────┘ │ │ insert_order() │ │ hash_password() │ │\n │ │ └────────┬─────────┘ └──────────┬────────────┘ │\n │ │ │ │ │\n │ │ │ SECURITY DEFINER │ triggers │\n │ │ ▼ ▼ │\n │ │ ┌──────────────────────────────────────────────┐ │\n │ │ │ data schema │ │\n BLOCKED │ │──────────────────────────────────────────────│ │\n │ │ │ customers orders ... │ │\n └ ─ ─ ─ ✕ │ └──────────────────────────────────────────────┘ │\n │ │\n └──────────────────────────────────────────────────────┘","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Skill Contents","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"🚀 Getting Started (Read These First)","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Document","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Purpose","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"quick-reference.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/quick-reference.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"QUICK LOOKUP","type":"text","marks":[{"type":"strong"}]},{"text":" - Single-page cheat sheet (print this!)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"schema-architecture.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/schema-architecture.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"START HERE","type":"text","marks":[{"type":"strong"}]},{"text":" - Schema separation pattern (data/private/api)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"coding-standards-trivadis.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/coding-standards-trivadis.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Coding standards & naming conventions (l_, g_, co_)","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"📚 Core Reference (Use Daily)","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Document","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Purpose","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"plpgsql-table-api.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/plpgsql-table-api.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Table API functions, procedures, triggers","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"schema-naming.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/schema-naming.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Naming conventions for all objects","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"data-types.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/data-types.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Data type selection (UUIDv7, text, timestamptz)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"indexes-constraints.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/indexes-constraints.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Index types, strategies, constraints","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"migrations.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/migrations.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Native migration system documentation","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"anti-patterns.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/anti-patterns.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Common mistakes to avoid","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"checklists-troubleshooting.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/checklists-troubleshooting.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Project checklists & problem solutions","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"🔧 Advanced Topics (When Needed)","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Document","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Purpose","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"testing-patterns.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/testing-patterns.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"pgTAP unit testing, test factories","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"performance-tuning.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/performance-tuning.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"EXPLAIN ANALYZE, query optimization, JIT","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"row-level-security.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/row-level-security.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"RLS patterns, multi-tenant isolation","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"jsonb-patterns.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/jsonb-patterns.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"JSONB indexing, queries, validation","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"audit-logging.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/audit-logging.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Generic audit triggers, change tracking","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"bulk-operations.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/bulk-operations.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"COPY, batch inserts, upserts","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"session-management.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/session-management.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Session variables, connection pooling","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"transaction-patterns.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/transaction-patterns.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Isolation levels, locking, deadlock prevention","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"full-text-search.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/full-text-search.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"tsvector, tsquery, ranking, multi-language","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"partitioning.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/partitioning.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Range, list, hash partitioning strategies","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"window-functions.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/window-functions.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Frames, ranking, running calculations","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"time-series.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/time-series.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Time-series data patterns, BRIN indexes","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"event-sourcing.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/event-sourcing.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Event store, projections, CQRS","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"queue-patterns.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/queue-patterns.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Job queues, SKIP LOCKED, LISTEN/NOTIFY","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"encryption.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/encryption.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"pgcrypto, column encryption, TLS","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"vector-search.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/vector-search.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"pgvector, embeddings, similarity search","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"postgis-patterns.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/postgis-patterns.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Spatial data, geographic queries","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"🚀 DevOps & Migration","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Document","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Purpose","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"oracle-migration-guide.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/oracle-migration-guide.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PL/SQL to PL/pgSQL conversion","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"cicd-integration.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/cicd-integration.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"GitHub Actions, GitLab CI, Docker","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"monitoring-observability.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/monitoring-observability.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"pg_stat_statements, metrics, alerting","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"backup-recovery.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/backup-recovery.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"pg_dump, pg_basebackup, PITR","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"replication-ha.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/replication-ha.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Streaming/logical replication, failover","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"📊 Data Warehousing","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Document","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Purpose","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"data-warehousing-medallion.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/data-warehousing-medallion.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Medallion Architecture","type":"text","marks":[{"type":"strong"}]},{"text":" - Bronze/Silver/Gold, data lineage, ETL","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"analytical-queries.md","type":"text","marks":[{"type":"link","attrs":{"href":"references/analytical-queries.md","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Analytical query patterns, OLAP optimization, GROUPING SETS","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Executable Scripts","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Script","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Purpose","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"001_install_migration_system.sql","type":"text","marks":[{"type":"link","attrs":{"href":"scripts/001_install_migration_system.sql","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Install migration system (core functions)","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"002_migration_runner_helpers.sql","type":"text","marks":[{"type":"link","attrs":{"href":"scripts/002_migration_runner_helpers.sql","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Helper procedures (","type":"text"},{"text":"run_versioned","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"run_repeatable","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"003_example_migrations.sql","type":"text","marks":[{"type":"link","attrs":{"href":"scripts/003_example_migrations.sql","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Example migration patterns","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"999_uninstall_migration_system.sql","type":"text","marks":[{"type":"link","attrs":{"href":"scripts/999_uninstall_migration_system.sql","title":null}}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Clean removal of migration system","type":"text"}]}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Core Architecture","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Schema Separation Pattern","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"Application → api schema → data schema\n ↓\n private schema (triggers, helpers)","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Schema","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Contains","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Access","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Purpose","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"data","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Tables, indexes","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"None","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Data storage","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"private","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Triggers, helpers","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"None","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Internal logic","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"api","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Functions, procedures","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Applications","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"External interface","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"app_audit","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Audit tables","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Admins","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Change tracking","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"app_migration","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Migration tracking","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Admins","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Schema versioning","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Security Model","type":"text"}]},{"type":"paragraph","content":[{"text":"All ","type":"text"},{"text":"api","type":"text","marks":[{"type":"code_inline"}]},{"text":" functions MUST have:","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"sql"},"content":[{"text":"SECURITY DEFINER\nSET search_path = data, private, pg_temp","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Quick Reference","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Create Table Pattern","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"sql"},"content":[{"text":"CREATE TABLE data.{table_name} (\n id uuid PRIMARY KEY DEFAULT uuidv7(),\n -- columns...\n created_at timestamptz NOT NULL DEFAULT now(),\n updated_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE TRIGGER {table}_bu_updated_trg\n BEFORE UPDATE ON data.{table_name}\n FOR EACH ROW EXECUTE FUNCTION private.set_updated_at();","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"API Function Pattern","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"sql"},"content":[{"text":"CREATE FUNCTION api.{action}_{entity}(in_param type)\nRETURNS TABLE (col1 type, col2 type)\nLANGUAGE sql STABLE\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\n SELECT col1, col2 FROM data.{table} WHERE ...;\n$;","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"API Procedure Pattern","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"sql"},"content":[{"text":"CREATE PROCEDURE api.{action}_{entity}(\n in_param type,\n INOUT io_id uuid DEFAULT NULL\n)\nLANGUAGE plpgsql\nSECURITY DEFINER\nSET search_path = data, private, pg_temp\nAS $\nBEGIN\n INSERT INTO data.{table} (...) VALUES (...) RETURNING id INTO io_id;\nEND;\n$;","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Migration Pattern","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":"sql"},"content":[{"text":"SELECT app_migration.acquire_lock();\n\nCALL app_migration.run_versioned(\n in_version := '001',\n in_description := 'Description',\n in_sql := $mig$ ... $mig$,\n in_rollback_sql := '...'\n);\n\nSELECT app_migration.release_lock();","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Naming Conventions","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Trivadis-Style Variable Prefixes","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Prefix","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Type","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Example","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"l_","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Local variable","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"l_customer_count","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"g_","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Session/global variable","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"g_current_user_id","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"co_","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Constant","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"co_max_retries","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"in_","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"IN parameter","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"in_customer_id","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"out_","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"OUT parameter (functions only)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"out_total","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"io_","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"INOUT parameter (procedures)","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"io_id","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"c_","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Cursor","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"c_active_orders","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"r_","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Record","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"r_customer","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"t_","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Array/table","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"t_order_ids","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"e_","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Exception","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"e_not_found","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"Note","type":"text","marks":[{"type":"strong"}]},{"text":": PostgreSQL procedures only support INOUT parameters, not OUT. Use ","type":"text"},{"text":"io_","type":"text","marks":[{"type":"code_inline"}]},{"text":" prefix for all procedure output parameters.","type":"text"}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Database Objects","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Object","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Pattern","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Example","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Table","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"snake_case","type":"text","marks":[{"type":"code_inline"}]},{"text":", plural","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"orders","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"order_items","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Column","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"snake_case","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"customer_id","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"created_at","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Primary Key","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"id","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"id","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Foreign Key","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"{table_singular}_id","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"customer_id","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Index","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"{table}_{cols}_idx","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"orders_customer_id_idx","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Unique","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"{table}_{cols}_key","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"users_email_key","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Function","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"{action}_{entity}","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"get_customer","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"select_orders","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Procedure","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"{action}_{entity}","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"insert_order","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"update_status","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Trigger","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"{table}_{timing}{event}_trg","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"orders_bu_trg","type":"text","marks":[{"type":"code_inline"}]}]}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Data Type Recommendations","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Use","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Instead Of","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"text","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"char(n)","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"varchar(n)","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"numeric(p,s)","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"money","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"float","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"timestamptz","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"timestamp","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"boolean","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"integer","type":"text","marks":[{"type":"code_inline"}]},{"text":" flags","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"uuidv7()","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"serial","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"uuid_generate_v4()","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"GENERATED ALWAYS AS IDENTITY","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"serial","type":"text","marks":[{"type":"code_inline"}]},{"text":", ","type":"text"},{"text":"bigserial","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"jsonb","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"json","type":"text","marks":[{"type":"code_inline"}]},{"text":", EAV pattern","type":"text"}]}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Critical Anti-Patterns","type":"text"}]},{"type":"ordered_list","attrs":{"order":1,"listStyle":"number"},"content":[{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"❌ Direct table access from applications","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"❌ ","type":"text"},{"text":"RETURNS SETOF table","type":"text","marks":[{"type":"code_inline"}]},{"text":" (exposes all columns)","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"❌ Missing ","type":"text"},{"text":"SET search_path","type":"text","marks":[{"type":"code_inline"}]},{"text":" with ","type":"text"},{"text":"SECURITY DEFINER","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"❌ ","type":"text"},{"text":"timestamp","type":"text","marks":[{"type":"code_inline"}]},{"text":" without timezone","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"❌ ","type":"text"},{"text":"NOT IN","type":"text","marks":[{"type":"code_inline"}]},{"text":" with subqueries (use ","type":"text"},{"text":"NOT EXISTS","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"❌ ","type":"text"},{"text":"BETWEEN","type":"text","marks":[{"type":"code_inline"}]},{"text":" with timestamps (use ","type":"text"},{"text":">= AND \u003c","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"❌ Missing indexes on foreign keys","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"❌ ","type":"text"},{"text":"serial","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"bigserial","type":"text","marks":[{"type":"code_inline"}]},{"text":" (use ","type":"text"},{"text":"IDENTITY","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"❌ ","type":"text"},{"text":"varchar(n)","type":"text","marks":[{"type":"code_inline"}]},{"text":" arbitrary limits (use ","type":"text"},{"text":"text","type":"text","marks":[{"type":"code_inline"}]},{"text":")","type":"text"}]}]},{"type":"list_item","content":[{"type":"paragraph","content":[{"text":"❌ ","type":"text"},{"text":"SELECT FOR UPDATE","type":"text","marks":[{"type":"code_inline"}]},{"text":" without ","type":"text"},{"text":"NOWAIT","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"SKIP LOCKED","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"PostgreSQL 18+ Features","type":"text"}]},{"type":"table","attrs":{"layout":null},"content":[{"type":"tr","content":[{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Feature","type":"text"}]}]},{"type":"th","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Usage","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"uuidv7()","type":"text","marks":[{"type":"code_inline"}]}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"id uuid DEFAULT uuidv7()","type":"text","marks":[{"type":"code_inline"}]},{"text":" - timestamp-ordered UUIDs","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Virtual generated columns","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"col type GENERATED ALWAYS AS (expr)","type":"text","marks":[{"type":"code_inline"}]},{"text":" - computed at query time","type":"text"}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"OLD","type":"text","marks":[{"type":"code_inline"}]},{"text":"/","type":"text"},{"text":"NEW","type":"text","marks":[{"type":"code_inline"}]},{"text":" in RETURNING","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"UPDATE ... RETURNING OLD.col, NEW.col","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Temporal constraints","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"PRIMARY KEY (id) WITHOUT OVERLAPS","type":"text","marks":[{"type":"code_inline"}]}]}]}]},{"type":"tr","content":[{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"NOT VALID","type":"text","marks":[{"type":"code_inline"}]},{"text":" constraints","type":"text"}]}]},{"type":"td","attrs":{"colspan":1,"rowspan":1,"colwidth":null,"alignment":""},"content":[{"type":"paragraph","content":[{"text":"Add constraints without full table scan","type":"text"}]}]}]}]},{"type":"hr","attrs":{"markup":"---"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"File Organization","type":"text"}]},{"type":"code_block","attrs":{"wrap":false,"language":""},"content":[{"text":"db/\n├── migrations/\n│ ├── V001__create_schemas.sql\n│ ├── V002__create_tables.sql\n│ └── repeatable/\n│ ├── R__private_triggers.sql\n│ └── R__api_functions.sql\n├── schemas/\n│ ├── data/ # Table definitions\n│ ├── private/ # Internal functions\n│ └── api/ # External interface\n└── seeds/ # Reference data","type":"text"}]},{"type":"hr","attrs":{"markup":"---"}}]},"metadata":{"date":"2026-06-05","name":"postgresql-best-practices","author":"@skillopedia","source":{"stars":3,"repo_name":"postgresql-best-practices","origin_url":"https://github.com/wimolivier/postgresql-best-practices/blob/HEAD/SKILL.md","repo_owner":"wimolivier","body_sha256":"b3738d9bbfecd439d532abc6464831a18c66da1eb62829a284f88f263acc4554","cluster_key":"dd7be0b39dcc26152ac128a8c7878fca2d5378b1b19e20afc814ee4bdefd72e6","clean_bundle":{"format":"clean-skill-bundle-v1","source":"wimolivier/postgresql-best-practices/SKILL.md","attachments":[{"id":"602bf4b9-a2e3-5e72-90c7-fdbf42e291f6","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/602bf4b9-a2e3-5e72-90c7-fdbf42e291f6/attachment.md","path":"CLAUDE.md","size":4350,"sha256":"417aa51d2fe20bd388948a3eff24b9ef647b806de4e94d838a4591d47a554b21","contentType":"text/markdown; charset=utf-8"},{"id":"be8753cb-3660-5e6b-a391-2a42836ce10c","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/be8753cb-3660-5e6b-a391-2a42836ce10c/attachment.md","path":"README.md","size":5722,"sha256":"0186b3f136975642c655c5887d1c8aef539a8ab70084bcd56050ad592e7757df","contentType":"text/markdown; charset=utf-8"},{"id":"8130233f-6aa2-57af-9022-04d995730a02","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/8130233f-6aa2-57af-9022-04d995730a02/attachment.md","path":"references/analytical-queries.md","size":34030,"sha256":"ce7b767339a5cbe5cda341d7dae890d9c62482609ca9bd781bb3908bd1597f10","contentType":"text/markdown; charset=utf-8"},{"id":"bb7adb9b-cfdb-5e0a-8234-610ce6b3f80c","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/bb7adb9b-cfdb-5e0a-8234-610ce6b3f80c/attachment.md","path":"references/anti-patterns.md","size":17301,"sha256":"0bb1d7db0fa6b1b4648675a18e577d4aa2bc9f2bd7817531f4c0973072ae7c76","contentType":"text/markdown; charset=utf-8"},{"id":"24a748e6-0205-59af-b4bb-9f37d6f7d4ed","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/24a748e6-0205-59af-b4bb-9f37d6f7d4ed/attachment.md","path":"references/audit-logging.md","size":22510,"sha256":"3d011657a57f18761585cdaf6dd90f95f12af51972ca4b1335e025c1f4fdc5d6","contentType":"text/markdown; charset=utf-8"},{"id":"ddc3f674-d3ae-5035-9597-e8a69f33444a","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/ddc3f674-d3ae-5035-9597-e8a69f33444a/attachment.md","path":"references/backup-recovery.md","size":19733,"sha256":"4545c7b7ec9c0898df9e42eea0aeb490e67accd837fd720c7dea02cf042f3553","contentType":"text/markdown; charset=utf-8"},{"id":"8ad69326-f579-5f42-af70-ba72a34fbfcc","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/8ad69326-f579-5f42-af70-ba72a34fbfcc/attachment.md","path":"references/bulk-operations.md","size":18465,"sha256":"d67d51889b65b39f38b276c0d06362ddd4dd57d07df4322bba69a9ec915b761c","contentType":"text/markdown; charset=utf-8"},{"id":"4ceb5654-6f96-568f-90ec-c960f197e13f","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/4ceb5654-6f96-568f-90ec-c960f197e13f/attachment.md","path":"references/checklists-troubleshooting.md","size":4110,"sha256":"87c94199aa6c5862b7363b06d76351df4b631fb372435f714abf8160e1a035b1","contentType":"text/markdown; charset=utf-8"},{"id":"a48b27d6-caf8-5735-9a0d-0e5f87c773c0","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/a48b27d6-caf8-5735-9a0d-0e5f87c773c0/attachment.md","path":"references/cicd-integration.md","size":21235,"sha256":"ccaf770c9b7e6bf43ac09ccc04b01b7735b467ab7976bf21f21fbe715d23a6c0","contentType":"text/markdown; charset=utf-8"},{"id":"f425bfe0-e059-59ac-a430-0ec92e9a1f60","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/f425bfe0-e059-59ac-a430-0ec92e9a1f60/attachment.md","path":"references/coding-standards-trivadis.md","size":37082,"sha256":"774c939a6e2966fa943fe2d27636773bb48c5a59a2f267a5ff1606cdd29c992d","contentType":"text/markdown; charset=utf-8"},{"id":"a5a2d594-d62d-5272-9b24-f1eaa1d85826","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/a5a2d594-d62d-5272-9b24-f1eaa1d85826/attachment.md","path":"references/data-types.md","size":14480,"sha256":"51fc2ca1d050dcf9738a6a9f2f2da23972a9eeda02fe94966ba7a946ffdc9dbb","contentType":"text/markdown; charset=utf-8"},{"id":"c37d8174-36eb-5e9f-a57c-414b8484cbb6","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/c37d8174-36eb-5e9f-a57c-414b8484cbb6/attachment.md","path":"references/data-warehousing-medallion.md","size":74853,"sha256":"3eb859483d9bc6f702a237bd4f69949f14a3abe57d89ff7b0101c2bf745a2198","contentType":"text/markdown; charset=utf-8"},{"id":"9cf6373e-a64f-5739-afb1-fd67125e9da5","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/9cf6373e-a64f-5739-afb1-fd67125e9da5/attachment.md","path":"references/encryption.md","size":16186,"sha256":"3ad4e93b6bd04d2ee0ae8baa6a46be2436cc31722fb18ebcecd7018df6477ce4","contentType":"text/markdown; charset=utf-8"},{"id":"1fef6309-5824-5914-89c6-2f7238183b20","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/1fef6309-5824-5914-89c6-2f7238183b20/attachment.md","path":"references/event-sourcing.md","size":20197,"sha256":"d180d2c316371dfee3864feb8c5828144ee18fea5ada1d337710aecf5f304ead","contentType":"text/markdown; charset=utf-8"},{"id":"b3a66dcc-f771-5e4a-9bad-e296da711e6a","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/b3a66dcc-f771-5e4a-9bad-e296da711e6a/attachment.md","path":"references/full-text-search.md","size":22341,"sha256":"d542f08e8c8a596c9f8db15e6cab537fe31dcd3b03dd4ee7adfb5740bef71558","contentType":"text/markdown; charset=utf-8"},{"id":"b255a7bc-8fa3-5350-9e70-f86b902fd65e","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/b255a7bc-8fa3-5350-9e70-f86b902fd65e/attachment.md","path":"references/indexes-constraints.md","size":19192,"sha256":"d5047810535204ceecd4e1d845aa853c7f1df82076f1a18be4d9b1ac9141018f","contentType":"text/markdown; charset=utf-8"},{"id":"8e5b39ab-d134-522c-9047-50c621e08f38","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/8e5b39ab-d134-522c-9047-50c621e08f38/attachment.md","path":"references/jsonb-patterns.md","size":34291,"sha256":"b3dda8c7a42255f562965734df84f7eb9e2922434bce426d7cb495a03d90ec63","contentType":"text/markdown; charset=utf-8"},{"id":"6fa57091-2674-5376-a3f3-d76452ec1b46","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/6fa57091-2674-5376-a3f3-d76452ec1b46/attachment.md","path":"references/migrations.md","size":31592,"sha256":"200660f2834756385344cca88b2494f5df8c62943f17a0afad743219ab839751","contentType":"text/markdown; charset=utf-8"},{"id":"6a308b96-9832-547a-9d4c-44048b6a5770","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/6a308b96-9832-547a-9d4c-44048b6a5770/attachment.md","path":"references/monitoring-observability.md","size":27906,"sha256":"0b3633ebd0ddae672ac961ec01b77859b8c0c96729c7977986d8758e5f04b253","contentType":"text/markdown; charset=utf-8"},{"id":"1a62d36f-96a9-5c67-9282-633bdbb4eefe","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/1a62d36f-96a9-5c67-9282-633bdbb4eefe/attachment.md","path":"references/oracle-migration-guide.md","size":39372,"sha256":"a501a652f5a92146545e398a8c49934b2c7b69584bb748f4d4fe338d0536c15e","contentType":"text/markdown; charset=utf-8"},{"id":"4d40818e-eadb-58bc-a026-6b09a9df3ca9","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/4d40818e-eadb-58bc-a026-6b09a9df3ca9/attachment.md","path":"references/partitioning.md","size":23821,"sha256":"15ddb68f51e834cbe1e26d61cddd1f0f97e0d281371203214da8d92a007fc7fa","contentType":"text/markdown; charset=utf-8"},{"id":"d97c4e7b-dc01-519d-8ab6-9024d88ad1bb","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/d97c4e7b-dc01-519d-8ab6-9024d88ad1bb/attachment.md","path":"references/performance-tuning.md","size":25789,"sha256":"d1b3f61d0c9aee023982a1b4e122a97475a4a5e46fdfa016ee3ca60a1a4415a0","contentType":"text/markdown; charset=utf-8"},{"id":"bca6a9bf-b770-5d57-b23e-35c240b58eba","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/bca6a9bf-b770-5d57-b23e-35c240b58eba/attachment.md","path":"references/plpgsql-table-api.md","size":22001,"sha256":"bc5300b64231d5b51912ffd00e75231adb4f6b921401fe0983ff0c4c6323e43d","contentType":"text/markdown; charset=utf-8"},{"id":"6002b803-8b8b-5820-91e6-0cde4bed4f27","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/6002b803-8b8b-5820-91e6-0cde4bed4f27/attachment.md","path":"references/postgis-patterns.md","size":14170,"sha256":"d14610e37b607f44f86cff3db7da3d72792342a3abfd942b18c628b7f992b6fb","contentType":"text/markdown; charset=utf-8"},{"id":"5aa6c5b2-0c92-5975-9607-1b89630101c4","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/5aa6c5b2-0c92-5975-9607-1b89630101c4/attachment.md","path":"references/queue-patterns.md","size":16671,"sha256":"575564390bbf44a7a4fb18cd1dc8c9b08856f7760da9eebfbdfd02ef94696a6c","contentType":"text/markdown; charset=utf-8"},{"id":"c9b7e465-9f0f-5fc3-b60c-0c533816c188","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/c9b7e465-9f0f-5fc3-b60c-0c533816c188/attachment.md","path":"references/quick-reference.md","size":8376,"sha256":"0747dc0255e8c5b76d2d413dc33afaabe2dd243afdecd3e365e05e796dbb2438","contentType":"text/markdown; charset=utf-8"},{"id":"06fad5e8-1cb9-590a-a50d-dde52b3ca201","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/06fad5e8-1cb9-590a-a50d-dde52b3ca201/attachment.md","path":"references/replication-ha.md","size":18401,"sha256":"fdf5b1698d0ef1df73977de6d6791c1115edcfad06c9c8604eabc59688fe7864","contentType":"text/markdown; charset=utf-8"},{"id":"3d78f2d3-a403-5b9f-a7eb-d0bff8921d1d","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/3d78f2d3-a403-5b9f-a7eb-d0bff8921d1d/attachment.md","path":"references/row-level-security.md","size":21622,"sha256":"6f955c0829157b680420b734226c0f865f80d19ff8e9f33adc2782c51b23f50d","contentType":"text/markdown; charset=utf-8"},{"id":"d2125553-1354-505c-85b5-d080d2e78f19","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/d2125553-1354-505c-85b5-d080d2e78f19/attachment.md","path":"references/schema-architecture.md","size":17403,"sha256":"eafde77f2b127b5ddaab0840648997fe8234fb305df78103f5afe1c1a32071de","contentType":"text/markdown; charset=utf-8"},{"id":"420aeedf-9749-56cb-9b35-6788dbb49718","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/420aeedf-9749-56cb-9b35-6788dbb49718/attachment.md","path":"references/schema-naming.md","size":7611,"sha256":"8e22fa519eb93b5d9a9ec8c83423fda2066ebe7fcc33d91ab3aec113e7c97380","contentType":"text/markdown; charset=utf-8"},{"id":"f2f3f8ba-647c-5dc1-b200-1d2f17fe7a1a","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/f2f3f8ba-647c-5dc1-b200-1d2f17fe7a1a/attachment.md","path":"references/session-management.md","size":19278,"sha256":"a3d76ce5094259b2db0504e0c23b070befbae91051212503210f5f835c6d7293","contentType":"text/markdown; charset=utf-8"},{"id":"502559c2-7a5f-55bf-8122-35511bc2203c","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/502559c2-7a5f-55bf-8122-35511bc2203c/attachment.md","path":"references/testing-patterns.md","size":26606,"sha256":"e9f212ca2b650f21b3aaac8b2556e0243a601c319f5f1e1717c4235ff4e1884b","contentType":"text/markdown; charset=utf-8"},{"id":"28d611ec-97c5-5588-9591-a38ea1b5be5c","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/28d611ec-97c5-5588-9591-a38ea1b5be5c/attachment.md","path":"references/time-series.md","size":19246,"sha256":"c2d04ce1aae396e34bedd03899eae22c46c0edfeabd413630ddd1d1e46b7405f","contentType":"text/markdown; charset=utf-8"},{"id":"7ca050da-e381-5956-abfc-a6f64b321582","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/7ca050da-e381-5956-abfc-a6f64b321582/attachment.md","path":"references/transaction-patterns.md","size":17689,"sha256":"1184220a88786fcea40615299175f40ae4bba8976638504e8ad6bb7df85197d3","contentType":"text/markdown; charset=utf-8"},{"id":"5a000f6b-e3c6-5021-a163-debde7318a1e","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/5a000f6b-e3c6-5021-a163-debde7318a1e/attachment.md","path":"references/vector-search.md","size":18484,"sha256":"78e3ac2388a98dfa67463801377454d81b1ba32fe4e999a2ced9eb27955edee1","contentType":"text/markdown; charset=utf-8"},{"id":"be0a79e1-2793-531c-bf7d-e73a7e96e660","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/be0a79e1-2793-531c-bf7d-e73a7e96e660/attachment.md","path":"references/window-functions.md","size":13369,"sha256":"0c025c4f73360c37ae2a7869940640f1907239d1928b3e8bb131e805bae14eff","contentType":"text/markdown; charset=utf-8"},{"id":"f0a26595-bfb6-5b20-a549-1fdc6aaa3960","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/f0a26595-bfb6-5b20-a549-1fdc6aaa3960/attachment.sql","path":"scripts/001_install_migration_system.sql","size":24769,"sha256":"dff9df6d984cb68640cda49d358ed41655d3e074567fa3b49a7e9d0fb8be0f8e","contentType":"application/sql"},{"id":"7e35c4ba-7738-5dfc-ab8c-6817d4714d42","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/7e35c4ba-7738-5dfc-ab8c-6817d4714d42/attachment.sql","path":"scripts/002_migration_runner_helpers.sql","size":11463,"sha256":"a958f8fb89862ad3fae44fa331e2659628bf231b0bb594d3ed175c9094201c5b","contentType":"application/sql"},{"id":"1091e768-2f30-5f2e-ac41-4a782fd51a05","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/1091e768-2f30-5f2e-ac41-4a782fd51a05/attachment.sql","path":"scripts/003_example_migrations.sql","size":11990,"sha256":"f6e1d3655e77f3ce6480b489dab14ac714718a33947e3db8100c9b7c5f1e9ade","contentType":"application/sql"},{"id":"e7e2e5b6-62b6-5bb7-ab1e-ebbcb6f5d08d","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/e7e2e5b6-62b6-5bb7-ab1e-ebbcb6f5d08d/attachment.sql","path":"scripts/999_uninstall_migration_system.sql","size":1089,"sha256":"d855020732c77cadefb2f85011d8a693e8b98d40816acbc611ca14edf934ec58","contentType":"application/sql"},{"id":"c4454c93-d0aa-5a6d-95eb-5cfc0e474163","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/c4454c93-d0aa-5a6d-95eb-5cfc0e474163/attachment.md","path":"tests/README.md","size":8533,"sha256":"4252564e6bb087ac52a967268122bdab22cbfed04ab418a272b2911998580a3d","contentType":"text/markdown; charset=utf-8"},{"id":"44dbf6ac-8bc7-59be-9d44-8da59472785d","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/44dbf6ac-8bc7-59be-9d44-8da59472785d/attachment.env","path":"tests/config/test_config.env","size":808,"sha256":"3bf62738079f00f0027541ffd8ccfe20044196ed145d00ecabe3e113fb3c5ae4","contentType":"text/plain; charset=utf-8"},{"id":"d0edc115-6953-5532-9c91-04ef6da7af15","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/d0edc115-6953-5532-9c91-04ef6da7af15/attachment.sql","path":"tests/framework/assertions.sql","size":26416,"sha256":"2afcf344580306139c1eae809da4080e1edc67e6b0c351a17a8ee2bff2348677","contentType":"application/sql"},{"id":"4196dfdb-31a2-50f4-8189-efb623b57a71","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/4196dfdb-31a2-50f4-8189-efb623b57a71/attachment.sql","path":"tests/framework/test_helpers.sql","size":14162,"sha256":"cf0a7940d2accf0672bd569f3bf1effaf0da14e2dd1a8f488a0949076fdfe242","contentType":"application/sql"},{"id":"f4cf38a9-56a5-5d87-a3c3-cb200644b621","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/f4cf38a9-56a5-5d87-a3c3-cb200644b621/attachment.sql","path":"tests/framework/test_runner.sql","size":13847,"sha256":"08aa358b69364b3cf8851bda7fc0a6862b61035d88931ba7dbfcc380e407d71b","contentType":"application/sql"},{"id":"09728840-827a-5eaa-ac29-4beb6bf9fb66","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/09728840-827a-5eaa-ac29-4beb6bf9fb66/attachment.sql","path":"tests/integration/010_full_workflow_test.sql","size":12883,"sha256":"401b906dd60e604359d935efcf338cfb658669abe6085f1bbd1bfa65882c6730","contentType":"application/sql"},{"id":"8ef1562e-caf5-5ae0-b9f6-a5c17f515265","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/8ef1562e-caf5-5ae0-b9f6-a5c17f515265/attachment.sql","path":"tests/integration/020_concurrent_access_test.sql","size":8690,"sha256":"8b0f51584764725250218a5f48530748f8acc9bd4a7faa08f7d75f424aeee9ee","contentType":"application/sql"},{"id":"57a496d3-4f69-54f6-bdab-8c22754a8514","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/57a496d3-4f69-54f6-bdab-8c22754a8514/attachment.sql","path":"tests/integration/030_rls_workflow_test.sql","size":16983,"sha256":"91cb02c2cfb1b921db40ede44768ff9d9c6ee716b9bc96b0463ea9a1bf2a0867","contentType":"application/sql"},{"id":"934c696f-1e15-51b2-94a5-119cfee8fdf7","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/934c696f-1e15-51b2-94a5-119cfee8fdf7/attachment.sql","path":"tests/modules/01_migration_system/010_install_test.sql","size":7906,"sha256":"f4a57045577fd7cbfbbd8c3dce0c2cfcc2344f1ab631411bfe2673daa422a240","contentType":"application/sql"},{"id":"ba48a63d-ec89-5095-b7d2-34ebf2af6aac","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/ba48a63d-ec89-5095-b7d2-34ebf2af6aac/attachment.sql","path":"tests/modules/01_migration_system/020_locking_test.sql","size":7224,"sha256":"71de8e238c195a009e00b11ace7b4ef3aac98d1cb3697595db5d241296e76ede","contentType":"application/sql"},{"id":"e0991e75-1933-59b8-9d96-fbae140da34d","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/e0991e75-1933-59b8-9d96-fbae140da34d/attachment.sql","path":"tests/modules/01_migration_system/030_versioned_test.sql","size":10954,"sha256":"2b28e882a035c651065b9ee2c74c7e73b4f6ee71551eedc998689e13790f230c","contentType":"application/sql"},{"id":"b2f9485b-88a0-5b7e-adc5-c664e001e79c","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/b2f9485b-88a0-5b7e-adc5-c664e001e79c/attachment.sql","path":"tests/modules/01_migration_system/040_repeatable_test.sql","size":8982,"sha256":"b29d5d17b4c86ae050d3f0b9566221bf9594be3566ddac3ca9a930d09d15b5f7","contentType":"application/sql"},{"id":"551cc4bc-2d6d-53d8-a478-bae54df36c2e","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/551cc4bc-2d6d-53d8-a478-bae54df36c2e/attachment.sql","path":"tests/modules/01_migration_system/050_checksum_test.sql","size":9297,"sha256":"327dee6f29ecdf0d827f60db1b135a5429d2ed517e46563d30d510c7d2d7d9a4","contentType":"application/sql"},{"id":"df02682e-3777-595a-8331-2b66b50257b5","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/df02682e-3777-595a-8331-2b66b50257b5/attachment.sql","path":"tests/modules/01_migration_system/060_rollback_test.sql","size":12426,"sha256":"c30055bbcab8c8ed79cdb8c662cfde130a4676f3267f36e0efad9925d571ce19","contentType":"application/sql"},{"id":"6db80509-d78b-5406-8f86-e59c25b814f6","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/6db80509-d78b-5406-8f86-e59c25b814f6/attachment.sql","path":"tests/modules/01_migration_system/070_batch_test.sql","size":12295,"sha256":"de0d186bce5b719bf05977f8d6bbda3e3074686b3a24d67f72739058dc061fc5","contentType":"application/sql"},{"id":"1ed354ef-25f8-55d3-a986-c57046c1067c","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/1ed354ef-25f8-55d3-a986-c57046c1067c/attachment.sql","path":"tests/modules/01_migration_system/080_info_status_test.sql","size":10621,"sha256":"6864981d86ed6dcfb704113570f24ffa125a395a54b38091703e94d8fe51e8b7","contentType":"application/sql"},{"id":"b3e8e12f-c13f-5d50-bee6-7e5aaa7518d1","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/b3e8e12f-c13f-5d50-bee6-7e5aaa7518d1/attachment.sql","path":"tests/modules/01_migration_system/090_idempotent_constraints_test.sql","size":12004,"sha256":"fa6f7e2c4512fa2b86fdd2836ffcdc2fa738417b0abd74bc9baab42795bb7e8b","contentType":"application/sql"},{"id":"a9397b15-96bc-5c54-a20d-311497f16d89","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/a9397b15-96bc-5c54-a20d-311497f16d89/attachment.sql","path":"tests/modules/02_schema_architecture/010_three_schema_test.sql","size":9133,"sha256":"7799bf82aae7711ca86eb3ab6a7d563643aa6af77106a50004c89bd7e51d42ea","contentType":"application/sql"},{"id":"02d7109d-3b0f-554a-8e1f-a9e659c9011a","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/02d7109d-3b0f-554a-8e1f-a9e659c9011a/attachment.sql","path":"tests/modules/02_schema_architecture/020_security_definer_test.sql","size":8636,"sha256":"ef90344dc7b282677812729331de583aa8862b67f0cb5a1c0d370b985b02e55c","contentType":"application/sql"},{"id":"4f81abef-0cc0-5fc9-87fb-5d6d1a8da54c","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/4f81abef-0cc0-5fc9-87fb-5d6d1a8da54c/attachment.sql","path":"tests/modules/02_schema_architecture/030_role_permissions_test.sql","size":7116,"sha256":"1d3e558bec1d3a1f60b747cec905fa4f6872fa6bb2a36fab86659b7f20afe834","contentType":"application/sql"},{"id":"9bb37699-b468-55b9-9a84-91ad527b8dee","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/9bb37699-b468-55b9-9a84-91ad527b8dee/attachment.sql","path":"tests/modules/03_plpgsql_patterns/010_naming_conventions_test.sql","size":9182,"sha256":"f713c9a967f31042f4c0a3d22593da823eab6b0179480585413d424561af42b6","contentType":"application/sql"},{"id":"9cc7a812-0490-506c-b8a7-a116d5c29d2e","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/9cc7a812-0490-506c-b8a7-a116d5c29d2e/attachment.sql","path":"tests/modules/03_plpgsql_patterns/020_table_api_test.sql","size":12134,"sha256":"0ed6ca0931a62059da42d65790438a92c979b0b1791ecb5b3b1c2b20d0cd326f","contentType":"application/sql"},{"id":"f5aa2851-7899-5edd-a3c6-991baa731699","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/f5aa2851-7899-5edd-a3c6-991baa731699/attachment.sql","path":"tests/modules/03_plpgsql_patterns/030_trigger_test.sql","size":12317,"sha256":"dd7ae40d90c4afe6d6506249a81a8ce4f68c3e0825c61bfe6805a27eb21f37f5","contentType":"application/sql"},{"id":"b4b860ae-f299-5c46-9fc3-25d8f9aea918","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/b4b860ae-f299-5c46-9fc3-25d8f9aea918/attachment.sql","path":"tests/modules/03_plpgsql_patterns/040_error_handling_test.sql","size":8448,"sha256":"79c0ea5397a78d3226e0006a982860b1b598f54a2e795717f58e8c8b23df9567","contentType":"application/sql"},{"id":"02be010e-e45e-5386-ad9e-f857956600b3","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/02be010e-e45e-5386-ad9e-f857956600b3/attachment.sql","path":"tests/modules/03_plpgsql_patterns/050_prepared_statements_test.sql","size":10507,"sha256":"9c943cc399469429a07725b6d619797fa8e184785d5277d36f42a26de635023c","contentType":"application/sql"},{"id":"3263f819-af22-50a5-a921-b105109f033d","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/3263f819-af22-50a5-a921-b105109f033d/attachment.sql","path":"tests/modules/04_data_types/010_uuidv7_test.sql","size":8181,"sha256":"dc17635e0667159c7b202cedfad4c52647591e1eacf3698b15bb8e30bfc0c88f","contentType":"application/sql"},{"id":"124d57d8-3d1e-5b0d-898d-e8a88049b06d","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/124d57d8-3d1e-5b0d-898d-e8a88049b06d/attachment.sql","path":"tests/modules/04_data_types/020_timestamptz_test.sql","size":8953,"sha256":"af5b803f9a8d7dadc183f2a332f6f26d48f03b69bcd4dd70c757e26c4086652c","contentType":"application/sql"},{"id":"614d42a2-8e0c-5631-a787-a10c2d8d90c3","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/614d42a2-8e0c-5631-a787-a10c2d8d90c3/attachment.sql","path":"tests/modules/04_data_types/030_numeric_test.sql","size":7988,"sha256":"e86447a16b4710082f9ef47f9da54b359d378595b4f3faad12ec8c45ae5793dd","contentType":"application/sql"},{"id":"c9cc860d-8009-5943-addb-b0614da6b64e","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/c9cc860d-8009-5943-addb-b0614da6b64e/attachment.sql","path":"tests/modules/04_data_types/040_jsonb_test.sql","size":9224,"sha256":"ae8bf1faa41a995b28fbf8f12e422a9b3edcae808cf9db3a2447a7fc5a28e88f","contentType":"application/sql"},{"id":"f7dd4267-e429-5f5c-8e89-9ff7d9f07d71","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/f7dd4267-e429-5f5c-8e89-9ff7d9f07d71/attachment.sql","path":"tests/modules/05_anti_patterns/010_demonstrates_correct_patterns.sql","size":12465,"sha256":"f1bb06f281f61802f3a7e122074f0d9c4b51f0f8461bb352b1bb75cf0d056961","contentType":"application/sql"},{"id":"3ca9f772-d52a-56d2-b7c5-494972188c05","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/3ca9f772-d52a-56d2-b7c5-494972188c05/attachment.sql","path":"tests/modules/05_anti_patterns/020_fk_index_detection_test.sql","size":11138,"sha256":"10bfa7816e31cd642d27e5b773256cf38aaad8a09fd11a135c2558a64cb0be89","contentType":"application/sql"},{"id":"4b2c6795-bab4-525b-b0b9-ad5ec9e09644","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/4b2c6795-bab4-525b-b0b9-ad5ec9e09644/attachment.sql","path":"tests/modules/05_anti_patterns/030_n_plus_one_test.sql","size":13381,"sha256":"4411909e1bd0947fa9ae05a328aeb7c4885dd929fd48fb9e1e7aeca7ba6c33f2","contentType":"application/sql"},{"id":"63ed173c-c5db-5322-842b-19756bdc8e06","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/63ed173c-c5db-5322-842b-19756bdc8e06/attachment.sql","path":"tests/modules/06_row_level_security/010_enable_rls_test.sql","size":10473,"sha256":"e55c553578af495dc92a9f60b9850b920fa54b3c1f47035b23001cb5bffeac4f","contentType":"application/sql"},{"id":"ff667b21-2eb0-562e-b5ce-fcaaa3d39eac","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/ff667b21-2eb0-562e-b5ce-fcaaa3d39eac/attachment.sql","path":"tests/modules/06_row_level_security/020_permissive_policies_test.sql","size":9997,"sha256":"fd0de1617b860f58940403073b30253cda960a4e9de99708145c680523fa0802","contentType":"application/sql"},{"id":"ec27bc08-f320-500e-b65f-ad2d77854433","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/ec27bc08-f320-500e-b65f-ad2d77854433/attachment.sql","path":"tests/modules/06_row_level_security/030_restrictive_policies_test.sql","size":10481,"sha256":"bfc9a32f30dde99c2c510594277277f85d3a533b372115ff6c3326182a1603e5","contentType":"application/sql"},{"id":"90b4f540-3f0d-5000-bd50-1f960c634769","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/90b4f540-3f0d-5000-bd50-1f960c634769/attachment.sql","path":"tests/modules/06_row_level_security/040_multi_tenant_test.sql","size":17864,"sha256":"65fec48b4c625ddc6af802c247320da4d8119178eaa9bae06d5d79766a434797","contentType":"application/sql"},{"id":"3ef8b46a-a4d2-5ca5-b134-0b9e680157a5","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/3ef8b46a-a4d2-5ca5-b134-0b9e680157a5/attachment.sql","path":"tests/modules/06_row_level_security/050_rls_performance_test.sql","size":10326,"sha256":"b20d8c3fefe08742010fa6ccbb088f4b16ea124ff59e2d0df949a486b4c950d2","contentType":"application/sql"},{"id":"42030204-cc51-51dc-a219-318ed0ed06ed","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/42030204-cc51-51dc-a219-318ed0ed06ed/attachment.sql","path":"tests/modules/07_audit_logging/010_audit_schema_test.sql","size":13330,"sha256":"d9f599230d2e2448a74d8f0db97e7dc36e27959e0fd852bc45b5c1d53f0137c2","contentType":"application/sql"},{"id":"fa976370-39c0-533e-afc6-0491f00f84fc","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/fa976370-39c0-533e-afc6-0491f00f84fc/attachment.sql","path":"tests/modules/07_audit_logging/020_audit_trigger_test.sql","size":16830,"sha256":"fb7c7b95371bb30b1c14e4a6c8a16c152b3d3ced55105fd18c41c328a00a775e","contentType":"application/sql"},{"id":"f7e54cd7-ce4a-5021-93ce-d10fb67f3ccd","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/f7e54cd7-ce4a-5021-93ce-d10fb67f3ccd/attachment.sql","path":"tests/modules/07_audit_logging/030_audit_context_test.sql","size":15692,"sha256":"bc6d5c3cc953e4f876a7f485c19f9212e07ad5d483ce0e9033e42e8f4702bc3b","contentType":"application/sql"},{"id":"c6ece03c-ad77-500c-bd37-932cc613b0cc","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/c6ece03c-ad77-500c-bd37-932cc613b0cc/attachment.sql","path":"tests/modules/08_medallion_architecture/010_bronze_layer_test.sql","size":13983,"sha256":"450f3b51c95ae2b208b528033cefedd1cac34d5e98f5d75513307b86519df8e8","contentType":"application/sql"},{"id":"38d0f774-39ff-539f-853f-46a6f382f2ba","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/38d0f774-39ff-539f-853f-46a6f382f2ba/attachment.sql","path":"tests/modules/08_medallion_architecture/020_silver_layer_test.sql","size":17208,"sha256":"7714bbdbb30f96dff868267fe4eebff63161d2c9857e59454851c434ba09230c","contentType":"application/sql"},{"id":"54151377-7999-5700-9ecf-a52ffd9d7617","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/54151377-7999-5700-9ecf-a52ffd9d7617/attachment.sql","path":"tests/modules/08_medallion_architecture/030_gold_layer_test.sql","size":17043,"sha256":"4d3a79777310dcece3eafc67dec4530e25993894c6a9d78dcd847f88ec820c66","contentType":"application/sql"},{"id":"1e33a5b1-de0a-5543-918a-72a4c84d55c6","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/1e33a5b1-de0a-5543-918a-72a4c84d55c6/attachment.sql","path":"tests/modules/08_medallion_architecture/040_lineage_test.sql","size":13867,"sha256":"6c37eef94d4033eb32af4d275ce511c35efd41eb01bb2b1b8939b561ba2d63c3","contentType":"application/sql"},{"id":"ace1f66e-d8db-5b44-adfc-f720fa3d3604","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/ace1f66e-d8db-5b44-adfc-f720fa3d3604/attachment.sql","path":"tests/modules/09_advanced_indexing/010_gin_index_test.sql","size":13977,"sha256":"7b12f28c791eb5d357be493e68a9cc4f89ac211f89cb53eab54e41e7e6c211e3","contentType":"application/sql"},{"id":"b95c1e03-7460-5141-bc48-71638ba64625","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/b95c1e03-7460-5141-bc48-71638ba64625/attachment.sql","path":"tests/modules/09_advanced_indexing/020_partial_index_test.sql","size":12340,"sha256":"744f70ce8503b071d19e52941c9f0a01f584257efaccfad5f8c0285ae3ba696e","contentType":"application/sql"},{"id":"1c2bf147-4c3e-55c9-971d-dce00412355d","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/1c2bf147-4c3e-55c9-971d-dce00412355d/attachment.sql","path":"tests/modules/09_advanced_indexing/030_covering_index_test.sql","size":12860,"sha256":"4254921db76987b20cc8f89eb401ea724200491d5660d88af69479e20d17c70e","contentType":"application/sql"},{"id":"49b2429b-f301-56a7-9b74-c9cac3dcf1d8","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/49b2429b-f301-56a7-9b74-c9cac3dcf1d8/attachment.sql","path":"tests/modules/09_advanced_indexing/040_concurrent_index_test.sql","size":12323,"sha256":"80eab4c061d97177f0b2b724cd9097d2e5149fe5f6035f0582acdf5181b33943","contentType":"application/sql"},{"id":"bad4dcc5-1442-523a-a3e4-19b8695e1f0f","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/bad4dcc5-1442-523a-a3e4-19b8695e1f0f/attachment.sql","path":"tests/modules/10_bulk_operations/010_upsert_test.sql","size":16669,"sha256":"0e34a113860926e4cae4e6e3ce14b22571bcf0ad01945fc3c1e60b1ce3d8fb17","contentType":"application/sql"},{"id":"26455ba9-939b-5101-b941-e79bfeb51166","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/26455ba9-939b-5101-b941-e79bfeb51166/attachment.sql","path":"tests/modules/10_bulk_operations/020_batch_insert_test.sql","size":14439,"sha256":"d65905ba8f2199ecd14181fced6fa475d00da3bb3d2576d1b0b98d4267e82a36","contentType":"application/sql"},{"id":"f3e135f7-f60d-5691-beb1-ba98705d63bc","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/f3e135f7-f60d-5691-beb1-ba98705d63bc/attachment.sql","path":"tests/modules/11_transaction_patterns/010_isolation_test.sql","size":10377,"sha256":"90bab3456049b240f20e9232eb6446b898fe9ed437cd36c26319b5f89071eebd","contentType":"application/sql"},{"id":"2a859310-90ab-5852-b61f-d3901f4eb1d7","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/2a859310-90ab-5852-b61f-d3901f4eb1d7/attachment.sql","path":"tests/modules/11_transaction_patterns/020_locking_test.sql","size":12283,"sha256":"9d6f4205c5a170409117278b20a9bc17472d024999504b8f208c22e59725af9c","contentType":"application/sql"},{"id":"8117f80f-aa1c-55d8-aad9-bffe4a09b421","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/8117f80f-aa1c-55d8-aad9-bffe4a09b421/attachment.sh","path":"tests/scripts/ci_runner.sh","size":4625,"sha256":"e2a36a6565bdceb968d64019a162ffe8a4aea39f534631cb27c9be77b0b127b2","contentType":"application/x-sh; charset=utf-8"},{"id":"28abe8df-908c-50cd-a3af-efb457816c0b","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/28abe8df-908c-50cd-a3af-efb457816c0b/attachment.sh","path":"tests/scripts/run_all_tests.sh","size":7214,"sha256":"b25da7dfdc982a317c3cd398e730244e8cd4e8171af48cdbd8c47df2b9607528","contentType":"application/x-sh; charset=utf-8"},{"id":"83730c87-8d51-5a48-a74c-a3b353186c1e","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/83730c87-8d51-5a48-a74c-a3b353186c1e/attachment.sh","path":"tests/scripts/run_module.sh","size":4178,"sha256":"9df8d71a8d49461cec1b627b6c39a535c386827d4a93da7edd6298185a252990","contentType":"application/x-sh; charset=utf-8"},{"id":"5cf97ed4-f55e-5c10-8c8c-8ed0f7a7eaaa","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/5cf97ed4-f55e-5c10-8c8c-8ed0f7a7eaaa/attachment.sql","path":"tests/setup/00_check_prerequisites.sql","size":3565,"sha256":"d3cb832c9cac522c68ab6a57b3aa88ce8277ca1f17b7914aa8dd3982d5a16dc5","contentType":"application/sql"},{"id":"c7e94fc7-0796-5e1c-b0a5-79b618f29fc3","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/c7e94fc7-0796-5e1c-b0a5-79b618f29fc3/attachment.sql","path":"tests/setup/01_install_test_framework.sql","size":2372,"sha256":"b659193214f161e2ef48cb8b85943a5940395b55f7665e8f24416170cadca242","contentType":"application/sql"},{"id":"cde88bb1-d17b-517c-b8f8-7ed8ce57bf22","key":"uploads/10433ee7-ad12-4ae0-b34e-97553e46c6c8/cde88bb1-d17b-517c-b8f8-7ed8ce57bf22/attachment.sql","path":"tests/teardown/01_cleanup.sql","size":5609,"sha256":"7bd6e1500e386caf4d14071028b0a0428b55ab57b9afef1e15cce217d81d567e","contentType":"application/sql"}],"bundle_sha256":"1d7e6076300b91d68597ca9b08845e03925603ff4e8c32908680a79d47a159d2","attachment_count":98,"text_attachments":98,"attachment_storage":"skillopedia-attachments-v1","binary_attachments":0,"excluded_attachments":[]},"cluster_size":1,"skill_md_path":"SKILL.md","import_metadata":{"date":"2026-06-05","author":"@skillopedia","version":"v1","category":"security","category_label":"Security"},"exact_dupes_collapsed_into_this":0},"version":"v1","category":"security","import_tag":"clean-skills-v1","description":"PostgreSQL 18+ enterprise best practices for database development.\n\nUSE THIS SKILL WHEN THE USER:\n- Creates schemas, tables, functions, procedures, or triggers\n- Writes PL/pgSQL code (naming conventions: l_, in_, io_, co_ prefixes)\n- Implements Table API (SECURITY DEFINER functions, schema separation)\n- Manages migrations, indexes, constraints, or query performance\n- Works with PostgreSQL 18+ features (uuidv7, virtual columns)\n- Builds Medallion Architecture data warehouses (Bronze/Silver/Gold)\n- Reviews code for anti-patterns or migrates from Oracle PL/SQL\n\nCORE PATTERNS:\n- Three-schema separation: data (tables) → private (internal) → api (external)\n- Table API: All access via SECURITY DEFINER with SET search_path\n- Native PL/pgSQL migration system (no Flyway/Liquibase needed)\n- Trivadis naming: l_ (local), in_ (input), io_ (inout), co_ (constant)\n","user-invocable":false}},"renderedAt":1782981283210}

Important: agents should read /llm.txt, /llms.txt, or /.well-known/skills.json to discover the public Skillopedia API.