The Operational Runbook for Declarative Syncs and Compliance

Your six-figure enterprise deal just hit a wall. Not because the prospect disliked the product demo, and not because a competitor undercut you on price. It died in procurement because the infosec team opened the vendor risk assessment questionnaire and found your integration layer listed as a stateful third-party data sub-processor.

An operational runbook for declarative syncs and compliance is a structured document that defines how your team configures, deploys, monitors, and audits configuration-driven data pipelines—without writing provider-specific code and without caching third-party data in a middleware layer. If you are an engineering leader or PM at a B2B SaaS company shipping integrations while passing SOC 2, HIPAA, or enterprise vendor risk assessments, this is the playbook you need.

When B2B SaaS companies move upmarket, their integration strategy must evolve. The custom Python scripts and Zapier templates that sustained your SMB tier will collapse under the weight of enterprise requirements. Enterprise buyers demand native, reliable connections to their customized systems, but they outright refuse to let their highly sensitive data sit in a third-party caching layer just so your engineering team can handle API retries.

This guide provides a step-by-step framework to move from brittle, hand-rolled sync scripts to declarative pipelines that are auditable, compliant, and operationally sane.

The Enterprise Integration Bottleneck

Moving upmarket exposes SaaS companies to a completely different class of security scrutiny. Procurement teams are hyper-vigilant. They scrutinize every node in your architecture that touches their data. The moment your integration middleware stores third-party customer data—even temporarily for retry buffers or workflow state—it becomes a data sub-processor in the eyes of procurement.

The cost of getting this wrong is severe. Breach costs increased 10% from the prior year, the largest yearly jump since the pandemic, as 70% of breached organizations reported that the breach caused significant or very significant disruption. The global average cost of a data breach reached $4.88 million in 2024. Enterprise infosec teams are hyper-aware of every third party that touches their data.

Simultaneously, integration capabilities are a strategic imperative to outperform competitors. 106 is the average number of SaaS apps per company in 2024, according to BetterCloud's State of SaaS report. Every one of those apps is a potential integration point your customers will ask about. The demand for integration has never been higher, and integration platform as a service is the largest integration platform market meeting this demand.

If each integration requires a dedicated sync script, and each script takes an engineer three weeks to build and harden, your integration backlog will outpace your hiring plan. To capture enterprise revenue without expanding headcount or failing SOC 2 audits, engineering teams must adopt declarative data sync pipelines built on zero-data-retention architecture.

Imperative Scripts vs. Declarative Data Pipelines

Most engineering teams start building integrations imperatively. A customer requests a Zendesk integration, so a developer writes a Node.js script. They implement Zendesk's specific cursor pagination, write a function to handle Zendesk's OAuth flow, and hardcode the mapping to your internal database schema.

Then a customer asks for Jira. The developer writes another script in Python, this time handling Jira's offset pagination, Basic Auth, and custom field structures. Six months later, you have a graveyard of brittle ETL scripts.

An imperative sync script tells the computer how to fetch data: write the HTTP client, handle pagination manually, manage OAuth refresh logic, serialize query parameters, parse responses, track cursors. The "how" is spread across hundreds of lines of code per provider.

The imperative approach (pseudocode)

# zendesk_sync.py - one of many provider-specific scripts
def sync_tickets(account, last_sync_date):
    token = refresh_oauth_token(account)  # you wrote this
    cursor = None
    while True:
        resp = requests.get(
            f"https://{account.subdomain}.zendesk.com/api/v2/tickets",
            headers={"Authorization": f"Bearer {token}"},
            params={"page[after]": cursor, "sort": "updated_at",
                    "filter[updated_at]": last_sync_date}
        )
        if resp.status_code == 429:
            time.sleep(int(resp.headers.get("Retry-After", 60)))
            continue
        data = resp.json()
        for ticket in data["tickets"]:
            normalized = map_zendesk_ticket(ticket)  # hand-written mapping
            emit_record(normalized)
        cursor = data.get("meta", {}).get("after_cursor")
        if not cursor:
            break

You will write a nearly identical file for ServiceNow, Freshdesk, and every other ticketing provider. When an upstream provider alters an endpoint, your team drops product work to push an emergency fix.

A declarative data pipeline fundamentally changes this dynamic. Instead of writing code that dictates what the end state should look like: which resources to sync, what the dependency graph looks like, and how to map fields between schemas.

The declarative approach (configuration)

{
  "integration_name": "zendesk",
  "resources": [
    {
      "resource": "ticketing/users",
      "method": "list"
    },
    {
      "resource": "ticketing/tickets",
      "method": "list",
      "query": {
        "updated_at": { "gt": "{{previous_run_date}}" }
      }
    },
    {
      "resource": "ticketing/comments",
      "method": "list",
      "depends_on": "ticketing/tickets",
      "query": {
        "ticket_id": "{{resources.ticketing.tickets.id}}"
      }
    }
  ]
}

In a declarative system, the runtime engine is a generic pipeline. It reads this JSON configuration describing how to talk to the API. Pagination strategy, auth handling, field normalization, and cursor tracking are all resolved from configuration data—not from code you wrote. Adding a new integration becomes a data operation, not a code deployment. Swapping Zendesk for Freshdesk means changing the integration_name field. For a deeper look at this architectural shift, read our guide on Declarative Data Sync Pipelines: Ship Integrations as Config, Not Code.

The Compliance Mandate: Why Zero Data Retention Wins

When you sell to healthcare, finance, or government buyers, your architecture will collide with their vendor risk assessment (VRA) process. Here is the part that kills enterprise deals: data residency in your integration middleware.

Many legacy iPaaS and unified API vendors rely on stateful architectures. They ingest data from the third-party API, store it in their own managed databases to normalize it, and then serve it to your application. This triggers a cascade of compliance liabilities:

• SOC 2 Confidentiality criteria require you to identify, retain, and dispose of confidential data according to documented policies. Under SOC 2, particularly within the Confidentiality and Availability Trust Services Criteria, having clear data retention policies is a critical expectation. Auditors want to see that your organization has thoughtfully determined how long data should be kept, why it's retained, and how it's securely disposed of when it's no longer needed.
• HIPAA mandates PHI retention for six years and strict access controls on any system that processes it.
• Enterprise VRAs will ask exactly where customer payload data is stored, who has access, and whether it can run inside their VPC.

If the integration vendor suffers a breach, your customer's data is exposed. Security teams understand this risk and will block deals if your integration layer retains sensitive payloads.

The alternative is a zero-data-retention architecture. In this model, the integration layer acts entirely as a stateless proxy. Data is fetched from the third-party API, transformed in memory using JSONata expressions, and streamed directly to your application or webhook endpoint. The payload is never written to disk by the middleware.

sequenceDiagram
    participant App as Your Application
    participant Proxy as Zero-Retention Pipeline
    participant API as Third-Party API
    
    App->>Proxy: Execute Sync Job (Declarative)
    Proxy->>API: Fetch Page 1 (OAuth injected in-flight)
    API-->>Proxy: Raw JSON Payload
    Note over Proxy: Transform in-memory<br>via JSONata mapping
    Proxy-->>App: Normalized Data Stream
    Note over Proxy: Payload discarded.<br>No data persisted to disk.

This architecture bypasses the heaviest VRA blockers. Because the integration layer does not store data at rest, the compliance burden shrinks dramatically.

For a complete breakdown of passing these security reviews, see our On-Prem Deployment & Compliance Guide for SaaS Integrations.

Building Your Operational Runbook for Declarative Syncs

Transitioning to a declarative pipeline requires a structured operational runbook. It is not a design document; it is the step-by-step playbook your team follows to configure, deploy, and maintain data syncs in production without writing custom code.

Step 1: Define Your Unified Schema

Before connecting to any third-party API, define the exact schema your application expects to receive. This isolates your core product from the chaos of third-party data models.

If you are syncing CRM contacts, your application should expect a standardized object regardless of whether the data comes from Salesforce, HubSpot, or Pipedrive.

Your schema should include:

The custom_fields escape hatch is non-negotiable. Enterprise Salesforce instances can have hundreds of custom fields, and your unified schema will never cover all of them. A good declarative system lets you customize mappings at the environment or even per-account level without changing the base schema. We cover this pattern in depth in our guide on shipping API connectors as data-only operations.

Step 2: Configure Field Mappings with JSONata

With your schema defined, configure the transformation expressions that convert each provider's response format into your common model. JSONata is a functional query and transformation language specifically designed for JSON. It is Turing-complete, side-effect free, and storable as a string in a database.

Instead of writing a JavaScript function to parse a HubSpot response, you write a JSONata expression.

HubSpot Response Mapping Example (YAML Config):

response_mapping: >-
  (
    {
      "id": response.id.$string(),
      "first_name": response.properties.firstname,
      "last_name": response.properties.lastname,
      "email_addresses": [
        response.properties.email ? { "email": response.properties.email, "is_primary": true }
      ]
    }
  )

For more complex mappings, such as advanced CRM contact extraction, a single JSONata expression handles field renaming, type coercion, array filtering, and custom field extraction:

response.{
  "id": Id.$string(),
  "first_name": FirstName,
  "last_name": LastName,
  "email_addresses": [{ "email": Email, "is_primary": true }],
  "phone_numbers": $filter([
    { "number": Phone, "type": "phone" },
    { "number": MobilePhone, "type": "mobile" }
  ], function($v) { $v.number }),
  "created_at": CreatedDate,
  "updated_at": LastModifiedDate,
  "custom_fields": $sift($, function($v, $k) { $k ~> /__c$/i and $boolean($v) })
}

This mapping is executed by the generic pipeline engine. If a customer needs a custom field mapped, you update the JSONata string for their specific environment. It can be versioned, overridden per customer, and hot-swapped. No code deployment is required.

Step 3: Set Up Incremental Sync Cursors

Syncing tens of thousands of records on every job run will exhaust API rate limits instantly. Full syncs are expensive. Once the initial data pull completes, subsequent runs should only fetch records that changed since the last successful run.

In a declarative system, this is typically a binding on a timestamp parameter within your sync job configuration:

{
  "resource": "ticketing/tickets",
  "method": "list",
  "query": {
    "updated_at": { "gt": "{{previous_run_date}}" }
  }
}

The runtime engine automatically tracks previous_run_date as the completion timestamp of the last successful sync run for that account. On the first run, it defaults to epoch (1970-01-01T00:00:00.000Z), effectively performing a full sync.

Document in your runbook:

• When to trigger a full re-sync: Schema changes, mapping updates, data corruption recovery.
• How cursors are scoped: Per sync job and per integrated account (not global).
• What happens if a run fails partway: The cursor should not advance; the next run retries from the same checkpoint.

Step 4: Define Resource Dependencies

Real-world syncs are rarely flat. Comments belong to tickets. Contacts belong to accounts. Your pipeline config needs to express these relationships declaratively:

{
  "resource": "ticketing/comments",
  "method": "list",
  "depends_on": "ticketing/tickets",
  "query": {
    "ticket_id": "{{resources.ticketing.tickets.id}}"
  }
}

This tells the runtime: for each ticket fetched in the ticketing/tickets step, fetch the associated comments. The depends_on field creates the dependency graph; the placeholder syntax dynamically injects parent record fields into child queries. Your runbook should document the dependency tree for each pipeline and the expected execution order.

Handling Rate Limits, Retries, and Pagination at Scale

Abstracting API communication sounds great in theory, but the operational realities of third-party APIs still apply. APIs go down, tokens expire, and rate limits are exceeded.

How Declarative Systems Handle Pagination

A resilient declarative engine abstracts pagination entirely. The pipeline config specifies the pagination strategy (cursor-based, offset-based, page-number, link header) and the response field that contains the next cursor. The runtime handles the loop. You never write while (cursor) { ... } again.

Different providers use wildly different pagination schemes. HubSpot uses cursor-based pagination with an after parameter. Salesforce uses SOQL query locators. Zendesk uses cursor-based page [after]. A good declarative system encodes these differences in the integration config, not in your application code, streaming the normalized data back to your application in a single continuous data stream.

Rate Limits: You Own the Backoff

Many unified API platforms claim to "handle" rate limits automatically by absorbing the errors and silently retrying on their own servers. This is a dangerous anti-pattern for enterprise systems. Silent retries mask underlying architectural flaws, cause unpredictable latency spikes, and often result in the middleware storing your data in a queue while it waits for the rate limit window to reset—violating zero-data-retention requirements.

A highly resilient architecture takes a radically honest approach: pass rate limit errors directly to the caller, but normalize the metadata.

When an upstream API returns HTTP 429 (Too Many Requests), your pipeline should pass that 429 error directly back to your application. However, because every API formats rate limit headers differently, the pipeline normalizes upstream rate limit info into standardized headers per the IETF spec:

• ratelimit-limit: The maximum number of requests permitted in the current window.
• ratelimit-remaining: The number of requests remaining in the current window.
• ratelimit-reset: The time at which the rate limit window resets (in UTC epoch seconds).

Your runbook should specify how your application reads these headers:

• Backoff strategy: Implement exponential backoff with jitter (typically base * 2^attempt + random(0, 1000)ms) in your application's state machine.
• Max retries: 3-5 for transient errors, 0 for auth failures.
• Alerting thresholds: Notify ops when a sync job hits rate limits more than N times in a single run.

Error Handling Modes

Declarative sync pipelines typically offer two error handling strategies that must be documented in your runbook:

1. Ignore and continue (default): Log the error, emit an error event, and proceed to the next resource. This is the right default for large syncs where a single 404 on one record shouldn't abort 10,000 successful ones.
2. Fail fast: Halt the entire pipeline on the first error. Use this for critical syncs (like financial reconciliation) where partial data is worse than no data.

Deploying and Monitoring Compliant Integrations

Once the declarative syncs are configured, the final phase of the runbook is operational monitoring and credential management.

Proactive OAuth Token Management

OAuth access tokens typically expire after 30 to 60 minutes. If a long-running sync job is executing when a token expires, the job will fail. Waiting for a 401 Unauthorized error before attempting a refresh is a reactive, fragile strategy.

Your operational runbook must mandate proactive token renewal. A well-designed system schedules token renewal ahead of expiry. For example, immediately after an OAuth token is acquired, the platform schedules work ahead of token expiry to fire an alarm 60 to 180 seconds before the token's exact expiration time. When the alarm fires, the system proactively negotiates a new access token.

Handling Concurrency with Mutex Locks

In enterprise environments, multiple sync jobs, webhooks, and user requests might attempt to use the same integrated account simultaneously. If the token expires, you risk a race condition where five concurrent processes all attempt to refresh the token at the exact same millisecond. This often triggers fraud-detection mechanisms at the upstream provider, resulting in a revoked refresh token and a disconnected customer.

To prevent this, the token refresh logic must be protected by a distributed mutex lock. When the first process detects an expired token (or the proactive alarm fires), it acquires a lock for that specific account ID. Subsequent concurrent requests see the lock and await the resolution of the first operation. Once the new token is acquired, the lock is released, and all pending requests proceed using the fresh credentials.

Your runbook should also cover retry policies for refresh failures. Retryable errors (5xx) get automatic multi-hour backoff; non-retryable errors (401 invalid_grant) stop retrying immediately, mark the account as needs_reauth, and surface a re-authentication prompt to the end user. For a deep dive on this topic, see our article on handling OAuth token refresh failures in production.

Monitoring Without Logging Sensitive Payloads

Enterprise compliance requires strict auditability. Security teams need to know exactly who connected an account, when a sync job ran, and what errors occurred. However, you cannot log the actual data payloads, or you violate the zero-data-retention policy.

Compliance-aware monitoring means your runbook must specify that the platform only logs metadata:

• Sync job run ID, start time, end time, and status.
• Target API endpoint and HTTP method.
• Record counts per resource (fetched, emitted, errored).
• Normalized error messages and HTTP status codes (extracted via JSONata error expressions, stripping out response bodies containing PII).
• Token refresh events and rate limit occurrences.

If an API returns a 400 Bad Request because a specific email address is malformed, the error expression should extract the structural reason for the failure without logging the actual email address into your observability stack.

Webhook Delivery for Real-Time Sync Events

Declarative sync pipelines emit structured webhook events that your application can consume in real-time:

Your application receives these events, writes the record data to your own datastore, and manages retention according to your own policies. The integration middleware never persists the data.

sequenceDiagram
    participant App as Your Application
    participant Engine as Sync Engine
    participant API as Third-Party API

    App->>Engine: Trigger sync run<br>(job_id, account_id)
    Engine->>API: Fetch page 1 (with auth, pagination)
    API-->>Engine: Response (records + cursor)
    Engine->>App: Webhook: sync_job_run:record<br>(normalized data)
    Engine->>API: Fetch page 2
    API-->>Engine: Response (records + cursor)
    Engine->>App: Webhook: sync_job_run:record<br>(normalized data)
    Engine->>App: Webhook: sync_job_run:completed
    Note over Engine: No data persisted<br>in middleware

Your Runbook Checklist

Before deploying any declarative sync pipeline to production, your runbook should have documented answers to each of the following:

• Unified schema defined for each resource category (CRM, HRIS, ticketing, etc.)
• Field mappings configured and tested for each provider you support
• Incremental sync cursors enabled with documented reset procedures
• Error handling mode specified per pipeline (ignore vs. fail-fast)
• Rate limit backoff strategy implemented in your webhook consumer
• OAuth token refresh monitored with alerting on needs_reauth events
• Webhook delivery endpoint deployed with idempotent record processing
• Monitoring dashboards showing sync run status, error rates, and latency
• Compliance documentation confirming zero data retention in the middleware layer
• Scheduled triggers configured (cron expressions for recurring syncs)
• Full re-sync procedure documented for disaster recovery scenarios

Scaling Integrations Without Scaling Headcount

The transition from imperative scripts to declarative data pipelines is not just a technical refactor; it is a strategic necessity for B2B SaaS companies moving upmarket.

Building all of this infrastructure from scratch—a generic execution engine, declarative configs for every provider, field mapping with a transformation language, pagination abstraction, token lifecycle management, and webhook delivery—would consume your engineering team for quarters. That is the practical argument for using a unified API platform that already implements this architecture.

Truto uses a generic execution engine driven entirely by JSON configuration and JSONata expressions. Adding a new provider is a data operation, not a code deployment. The platform acts as a pass-through proxy with zero data retention, which means your SOC 2 audit scope stays clean.

Enterprise buyers will not compromise on security, and they will not wait six months for your engineering team to build custom connectors. By adopting a zero-data-retention architecture, defining integrations purely as configuration data, and standardizing rate limit and authentication handling, you can scale your integration catalog exponentially.

You eliminate the maintenance burden of custom code, pass strict vendor risk assessments with ease, and allow your engineering team to focus on building your core product.