fix(persistence): chunk block and edge inserts to prevent sql variabl…

[Siddhartha-singh01]

Summary

Fixes the bug where workflows disappear or show up empty after a page refresh in self-hosted mode (SQLite).

The issue was that saveWorkflowToNormalizedTables was trying to bulk insert all blocks at once. SQLite has a hard limit of 999 parameters per query. Since every block has 17 fields, saving workflows with around 60 blocks was hitting the limit and silently failing the save transaction.

I just added a quick fix to chunk the array into smaller batches of 50 before inserting them.

Fixes #2424

[cursor]

PR Summary

Medium Risk
Changes the core workflow persistence path by splitting block/edge/subflow inserts into multiple statements inside the transaction; mistakes here could lead to partial or slower saves, but the logic is straightforward and scoped.

Overview
Prevents workflows from failing to persist on SQLite by chunking inserts in saveWorkflowToNormalizedTables so large workflows don’t exceed SQLite’s 999-parameter limit.

Blocks, edges, and subflows are now inserted in batches (CHUNK_SIZE = 50) instead of a single bulk insert, while keeping the same transactional delete-and-reinsert behavior; the rest of the changes are minor formatting/whitespace adjustments.

^{Written by Cursor Bugbot for commit 2e63582. This will update automatically on new commits. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Mar 6, 2026 2:16pm

[greptile-apps]

Greptile Summary

This PR fixes a critical data-loss bug in self-hosted (SQLite) deployments where workflows would disappear after a page refresh. The root cause was that saveWorkflowToNormalizedTables attempted to bulk-insert all blocks, edges, and subflows in a single SQL statement, hitting SQLite's hard limit of 999 bound parameters and causing the transaction to fail silently.

The fix introduces a CHUNK_SIZE = 50 constant and replaces the single bulk inserts with sequential chunked inserts inside the existing db.transaction, preserving full atomicity (any chunk failure rolls back the entire transaction).

Key points:

• With 17 fields per block, a chunk of 50 yields 850 parameters — safely under the 999-parameter ceiling.
• Edges (6 fields) and subflows (4 fields) are also chunked at 50, well under the limit.
• Chunking is applied globally (PostgreSQL and SQLite alike), introducing minimal overhead for non-SQLite deployments.
• The implementation is logically correct and transaction atomicity is fully preserved.

Style feedback:

• CHUNK_SIZE is currently defined inside the db.transaction callback; moving it to module scope would be cleaner and make the intent clearer.
• The derivation of chunk size should be documented with a comment, since if new fields are added to workflowBlocks without updating CHUNK_SIZE, the code will silently fail.

Confidence Score: 4/5

• The fix is logically correct and safe to merge; it properly handles SQLite's parameter limit constraints while preserving transaction atomicity.
• The core fix is solid—chunking block, edge, and subflow inserts prevents hitting SQLite's 999-parameter limit and fully preserves transaction atomicity. The only remaining feedback is stylistic: moving CHUNK_SIZE to module scope and documenting the chunk size derivation. These are minor improvements that don't affect correctness or safety.
• No files require special attention. The changes are isolated to saveWorkflowToNormalizedTables and are minimal in scope.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[saveWorkflowToNormalizedTables called] --> B[Begin db.transaction]
    B --> C[DELETE existing blocks / edges / subflows]
    C --> D{blocks.length > 0?}
    D -- Yes --> E[Build blockInserts array]
    E --> F[Loop: slice chunks of 50\ninsert each chunk into workflowBlocks]
    D -- No --> G{edges.length > 0?}
    F --> G
    G -- Yes --> H[Build edgeInserts array]
    H --> I[Loop: slice chunks of 50\ninsert each chunk into workflowEdges]
    G -- No --> J[Build subflowInserts array]
    I --> J
    J --> K{subflowInserts.length > 0?}
    K -- Yes --> L[Loop: slice chunks of 50\ninsert each chunk into workflowSubflows]
    K -- No --> M[Commit transaction]
    L --> M
    M --> N[Return success: true]

    style F fill:#d4edda,stroke:#28a745
    style I fill:#d4edda,stroke:#28a745
    style L fill:#d4edda,stroke:#28a745

Loading

_{Last reviewed commit: 160dc1c}

[greptile-apps]

CHUNK_SIZE is defined inside the db.transaction async callback, which means it's re-created on every call to saveWorkflowToNormalizedTables. Since it's a fixed, never-changing value, consider hoisting it to module scope. This makes the intent clearer and avoids unnecessary re-allocation.

Suggested change

	const CHUNK_SIZE = 50
	const CHUNK_SIZE = 50

Place this near the top of the file alongside other module-level constants (e.g., after the logger definition on line 22).

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[greptile-apps]

The SQLite hard limit of 999 bound parameters per statement is implicit in the choice of CHUNK_SIZE = 50, but this constraint is not documented. With 17 fields per block record, a chunk of 50 yields 850 parameters—safely under the limit. However, if new fields are ever added to workflowBlocks without updating CHUNK_SIZE, the code will silently fail. Consider documenting the derivation:

Suggested change

	for (let i = 0; i < blockInserts.length; i += CHUNK_SIZE) {
	await tx.insert(workflowBlocks).values(blockInserts.slice(i, i + CHUNK_SIZE))
	}
	// SQLite limits bound parameters to 999 per statement.
	// workflowBlocks has 17 fields → max safe chunk = floor(999/17) = 58.
	// Using 50 for a conservative margin.
	for (let i = 0; i < blockInserts.length; i += CHUNK_SIZE) {
	await tx.insert(workflowBlocks).values(blockInserts.slice(i, i + CHUNK_SIZE))
	}

[icecrasher321]

@Siddhartha-singh01 please point this to the staging branch not main

[Siddhartha-singh01]

Thanks for the feedback @icecrasher321! I've just pushed an update to include those comments explaining the chunking logic.
I'm also updating the base branch to staging now.