Update Procedures and Guarantees

When you update policies, domains, or users in Housecarl, you need to understand what guarantees the system provides and how changes propagate. This guide explains the transactional semantics, timing guarantees, and safe update procedures.

The Big Question: When Do Changes Take Effect?

Short answer: Immediately for most operations, with important caveats for JWT tokens.

Detailed answer: It depends on what you're updating. Let's break it down.

Policy Updates

When They Take Effect

Policy changes are immediately effective for all new authorization requests.

# You deploy a new policy (requires domain UUID)
housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 new-policy.toml

# Immediately after this returns successfully:
# - New authz requests use the new policy
# - In-flight requests continue with old policies (see below)
# - No service restart needed
# - No cache warming needed

ACID Guarantees for Policy Updates

Housecarl provides ACID (Atomicity, Consistency, Isolation, Durability) guarantees for policy updates:

Atomicity

Policy updates are atomic at the domain level. When you use put-policies:

# Deploy multiple policies atomically (requires domain UUID)
housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 policy1.toml policy2.toml policy3.toml

Either:

• All three policies are deployed successfully, OR
• None of them are deployed (system remains in previous state)

No partial updates. You never end up with only policy1 and policy2 deployed but policy3 missing.

Consistency

Policy updates maintain consistency invariants:

• Domain must exist before adding policies
• Policy names must be unique within a domain
• Policy syntax must be valid
• Tenant must be active

If any invariant is violated, the entire update is rejected:

$ housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 invalid-policy.toml
Error: PolicyValidationError: Invalid regex pattern in policy 'bad-regex-policy'
# No changes are made

Isolation

Concurrent policy updates are serialized at the domain level:

Scenario: Two admins update the same domain simultaneously:

• Admin A deploys policies at 14:00:00.000
• Admin B deploys policies at 14:00:00.001

Result: One update completes, then the other. The second update sees the effects of the first. The final state is the result of the second update (last write wins).

Scenario: Two admins update different domains:

• Admin A updates "engineering" domain
• Admin B updates "product" domain

Result: Updates happen in parallel with no blocking. Fully isolated.

Durability

Once housectl domain put-policies returns successfully:

• Changes are written to durable storage (PostgreSQL)
• Survive server crashes
• Survive process restarts
• No data loss

In-Flight Authorization Requests

Q: What happens to authorization requests that are being evaluated when I update policies?

A: In-flight requests continue with the policy snapshot they started with.

Timeline:

14:00:00.000 - Request A starts (using old policies)
14:00:00.100 - Policy update begins
14:00:00.200 - Policy update commits
14:00:00.300 - Request A completes (still using old policies)
14:00:00.400 - Request B starts (using new policies)

Why: Policy evaluation loads the policy set at the start of the request. This prevents race conditions where a policy changes mid-evaluation.

Practical impact: For a brief moment (milliseconds to seconds depending on request complexity), some requests use old policies while others use new policies. This is generally acceptable and prevents inconsistent evaluation states.

Policy Update Workflow

Here's the safe procedure for updating policies in production:

Step 1: Test in Development

# Switch to dev tenant
housectl config use dev-context

# Deploy policies to dev domain (get domain UUID first with: housectl domain list)
housectl domain put-policies 660e8400-e29b-41d4-a716-446655440001 new-policies/

# Test authorization
housectl authz can-i test-request.json

# Verify expected behavior

Step 2: Validate Policy Syntax

# Parse policies locally (fast validation)
housectl authz parse-policies new-policies/

# Expected output:
All policies are valid.

Step 3: Test Locally (Optional)

# Test policies offline without touching any server
housectl authz can-i-local --request test-request.json new-policies/policy1.toml

Step 4: Deploy to Production

# Switch to production tenant
housectl config use prod-context

# Get your production domain UUID
housectl domain list

# Atomic deployment (replace UUID with your production domain UUID)
housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 new-policies/

# Verify deployment (list-policies accepts name or UUID)
housectl domain list-policies production

Step 5: Monitor and Verify

After deploying:

• Monitor authorization deny rates (should not spike unexpectedly)
• Test a few authorization requests manually
• Check application logs for authorization failures
• Be ready to rollback if needed

Rolling Back Policy Changes

If you need to revert a policy change:

# Option 1: Redeploy the previous version (recommended)
# (Assumes you keep policy versions in git)
git checkout HEAD~1 policies/
housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 policies/

# Option 2: Remove the problematic policy
# (Deploy only the policies you want to keep)
housectl domain put-policies 550e8400-e29b-41d4-a716-446655440000 good-policy1.toml good-policy2.toml

Important: There's no "undo" button. You roll back by deploying the previous policy set. This is why version control (git) for policy files is critical.

Domain Updates

Creating Domains

Domain creation is immediate and atomic:

housectl domain create new-domain

Once this returns, the domain exists and can be used immediately.

Updating Domain Hierarchies

Changing superior domain relationships affects policy evaluation immediately:

# Add global as a superior for engineering
housectl domain add-superior engineering global

Impact:

• Immediately after this succeeds, authorization requests in the engineering domain will inherit policies from global
• In-flight requests continue with old hierarchy
• No gradual rollout or caching delay

Risk: Changing domain hierarchies can dramatically change authorization behavior. Test thoroughly before updating production hierarchies.

Deleting Domains

Domain deletion is permanent and immediate:

housectl domain delete old-domain --yes

What gets deleted:

• The domain itself
• All policies in the domain
• Superior domain relationships pointing to this domain

What is NOT deleted:

• Child domains (they become orphaned - no superior domain)
• Resources that referenced this domain (they become orphaned - no policies apply)

Effect on authorization:

• Requests to resources in the deleted domain will be DENIED (no policies match)
• Child domains lose inherited policies from this domain

Recommendation: Before deleting a domain:

1. Check if any domains inherit from it: housectl domain list and review superior relationships
2. Migrate child domains to a new superior
3. Ensure no resources are actively using this domain
4. Delete the domain in dev/staging first to validate impact

User Updates

Creating Users

User creation is immediate:

housectl user create alice Secret123! alice@example.com

Security note: Passing passwords as command-line arguments exposes them in shell history and process listings. In sensitive environments, disable shell history before running this command, or use a secrets manager to supply the value.

Alice can authenticate immediately after this returns.

Updating User Attributes

User attribute updates (email, username, active status) are immediate:

housectl user update alice --email newalice@example.com

However, there's an important caveat: JWT tokens.

The JWT Token Challenge

Housecarl uses JWT tokens for authentication. User attributes are embedded in the JWT when the user logs in:

{
  "user_id": "550e8400-e29b-41d4-a716-446655440000",
  "username": "alice",
  "tenant_id": "660e8400-e29b-41d4-a716-446655440001",
  "email": "alice@example.com",
  "exp": 1704153600
}

The problem: JWTs are stateless. Once issued, they're valid until they expire (typically 12 hours by default).

Scenario:

14:00 - Alice logs in, gets JWT token with email "alice@example.com"
14:30 - Admin updates Alice's email to "newalice@example.com"
15:00 - Alice makes an authz request using her JWT token
       - JWT still says "alice@example.com" (old email)
       - Database says "newalice@example.com" (new email)

Impact:

• Most authorization decisions use user_id, not email, so they're unaffected
• If policies match on email attribute, they'll see the old email until token expires
• Username changes have the same issue

Solution: After updating user attributes that policies might match on:

# Option 1: Ask user to log out and log back in
# New JWT will have updated attributes

# Option 2: Wait for token expiration (default: 12 hours)

# Option 3: Invalidate user's sessions (if supported)
# Force re-authentication

Best practice: Design policies to match on stable identifiers (user_id, tenant_id) rather than mutable attributes (email, username).

Changing Passwords

Password changes take effect immediately:

housectl user change-password alice NewSecret456!

Security note: Passing passwords as command-line arguments exposes them in shell history and process listings. In sensitive environments, disable shell history before running this command, or use a secrets manager to supply the value.

Important: Existing JWT tokens remain valid. Changing a password does NOT invalidate active sessions.

If you need to force re-authentication (e.g., suspected compromise):

1. Change the password
2. Wait for token expiration, OR
3. User logs out and logs back in with new password

Deactivating Users

Setting a user to inactive:

housectl user update alice --active false

Effect:

• User cannot log in (new authentications fail)
• Existing JWT tokens remain valid until expiration
• Authorization requests with valid JWTs continue to work

Recommendation: For immediate effect, implement token revocation checking in your application, or wait for token expiration.

Tenant Updates

Creating Tenants

Tenant creation is immediate and atomic:

housectl admin create "New Corp" "Description"

What gets created:

• Tenant record
• Root domain for the tenant
• Initial policies in root domain (grants creator full access)

All created atomically - either all succeed or none do.

Updating Tenant Metadata

Tenant name, description, subscription updates are immediate:

housectl tenant update new-corp --name "New Corporation Inc" --description "Updated description"

No impact on authorization decisions (metadata only).

Disabling Tenants

Disabling a tenant:

housectl admin disable tenant-id

Effect:

• All authorization requests for resources in this tenant are DENIED
• Users cannot authenticate against this tenant
• Existing JWT tokens for this tenant become invalid (checked on each request)

Immediate - takes effect on the next request after the disable operation completes.

Use case: Suspend a customer for non-payment, security incident, or account closure.

Re-enabling Tenants

housectl admin enable tenant-id

Immediate - tenant becomes active again, authorization resumes normally.

Tenant Association Updates

Adding User to Tenant

housectl tenant associate-user acme-corp alice

Effect:

• Alice can now authenticate to acme-corp tenant
• Alice can receive JWTs for acme-corp tenant
• Immediate effect for new logins
• Existing JWTs for other tenants unchanged

Removing User from Tenant

housectl tenant disassociate-user acme-corp alice

Effect:

• Alice cannot authenticate to acme-corp tenant
• Existing JWTs for acme-corp remain valid until expiration

Recommendation: After disassociating, wait for JWT expiration or implement token revocation.

Consistency Guarantees Across Concurrent Requests

Scenario: Reading Your Own Writes

Admin A: Updates policy in engineering domain at 14:00:00.100
Admin A: Immediately queries engineering domain at 14:00:00.200

Result: Admin A sees the updated policy

Housecarl guarantees read-your-writes consistency. After a write completes, subsequent reads by the same or different users see the updated state.

Scenario: Concurrent Authorization Requests

Time 0: Policy allows alice to read documents
Time 1: Admin updates policy to deny alice
Time 2: Alice request A (started at time 1.5, finishes at time 3)
Time 3: Alice request B (started at time 2.5, finishes at time 4)

Result:

• Request A might use old or new policy (started during transition)
• Request B uses new policy (started after update completed)

Guarantee: No request sees an inconsistent mix of old and new policies. Each request sees a consistent snapshot.

Scenario: Multiple Domains

Admin A: Updates engineering domain at time 1
Admin B: Updates product domain at time 1

Authorization request: Needs policies from both domains

Guarantee: The request sees a consistent snapshot of the policy database. Either:

• Both updates are visible, OR
• Neither update is visible, OR
• One update is visible (depending on exact timing)

Never: Sees a partial update of one domain (ACID atomicity guarantees this).

Monitoring Update Impact

After making updates, monitor these metrics:

Authorization Deny Rate

# Watch for unexpected spikes
# (Assumes you have metrics/observability set up)

# Manual testing:
housectl authz can-i test-request.json

An unexpected increase in denies might indicate:

• Policy too restrictive
• Policy pattern mismatch
• Missing policy after deletion

Application Error Logs

Check your application's authorization error logs:

2024-01-15 14:05:23 [WARN] Authorization denied for alice: read /documents/project-alpha/spec.md

Spike in authorization denials after a policy update indicates potential issues.

Latency

Policy updates should not significantly impact authorization request latency. If latency spikes after an update:

• Check if you added a very complex regex policy
• Verify database connection pool is healthy
• Check for lock contention (very rare)

Safe Update Checklist

Before updating policies in production:

• Test in development - Deploy to dev/staging tenant first
• Validate syntax - Run housectl authz parse-policies
• Test locally - Use housectl authz can-i-local for critical paths
• Version control - Commit policy files to git before deploying
• Have rollback plan - Keep previous version accessible
• Monitor impact - Watch deny rates and logs after deployment
• Communicate - Notify team of policy changes that might affect their work
• Document - Update policy documentation with rationale for changes

Before updating domain hierarchies:

• Map current hierarchy - Understand current superior relationships
• Identify impact - List all affected domains and policies
• Test in isolation - Create a test domain with new hierarchy
• Plan migration - Have step-by-step procedure
• Rollback plan - Know how to revert hierarchy changes
• Off-hours - Consider deploying during low-traffic periods

Before deactivating users or tenants:

• Confirm necessity - Verify the user/tenant should be deactivated
• Check dependencies - Identify what breaks when deactivated
• Communication - Notify affected parties if appropriate
• Wait for JWT expiration - Or implement token revocation
• Audit trail - Document why and when deactivation occurred

Performance Considerations

Policy Update Performance

Policy updates are fast for typical workloads. Time scales with the number of policies being deployed and includes:

• Parsing policy TOML
• Validating policy structure
• Database transaction
• Durability guarantee (PostgreSQL commit)

Authorization Request Performance During Updates

Authorization requests during a policy update:

• No blocking - Reads and writes don't block each other
• Snapshot isolation - Requests see consistent policy snapshot
• Typical latency - 1-10ms for authorization request (unaffected by updates)

Database Locking

Housecarl uses PostgreSQL row-level locking:

• Policy updates: Lock only the affected domain row
• Domain updates: Lock only the affected domain row
• User updates: Lock only the affected user row

Implication: Multiple admins can update different domains simultaneously with no contention.

Edge Cases and Gotchas

Gotcha 1: JWT Token Lag

Issue: User attributes in JWTs don't update until token expires.

Solution: Design policies to match on stable IDs, not mutable attributes.

Gotcha 2: Domain Deletion Orphans Children

Issue: Deleting a superior domain doesn't delete child domains; they lose inherited policies.

Solution: Before deleting, reassign child domains to a new superior or delete them explicitly.

Gotcha 3: Last Write Wins

Issue: Concurrent updates to the same domain result in last-write-wins. No merge.

Solution: Use version control, coordinate with team, or implement optimistic locking in your workflow.

Gotcha 4: No Gradual Rollout

Issue: Policy updates are all-or-nothing, no gradual rollout built-in.

Solution: For critical policy changes, use domain hierarchy:

• Create new domain with new policy
• Migrate resources incrementally to new domain
• Once validated, update original domain

Gotcha 5: Deny Policies Are Forever (Until Updated)

Issue: A deny policy overrides all allows. If you add a deny policy by mistake, all access is blocked immediately.

Solution:

• Test deny policies thoroughly
• Keep deny policies in separate, clearly-named files
• Document deny policies heavily

Summary: Update Timing

ACID Summary

Housecarl provides ACID guarantees for updates:

• Atomicity: Operations complete fully or not at all
• Consistency: Invariants are maintained (valid policies, existing domains, etc.)
• Isolation: Concurrent updates are serialized per resource (domain/user/tenant)
• Durability: Successful operations survive crashes and restarts

No distributed transaction coordination required - each operation is atomic within PostgreSQL.

Next Steps

• Learn about policies: Policy Administration
• Understand the domain model: Tenants, Domains, and Policies
• See policy testing guidance: Policy Administration
• Production integration patterns: Developer Cookbook

Key Takeaway: Housecarl updates are immediate and ACID-compliant, but JWT token lag is the primary exception. Design policies around stable identifiers and implement token revocation if you need immediate effect for user changes.