Sitelet https://github.com/simstudioai/sim/pull/7077
Skip to content

fix(knowledge): harden ingestion pipeline - #7077

Open
waleedlatif1 wants to merge 3 commits into
stagingfrom
codex/knowledge-ingestion-hardening
Open

fix(knowledge): harden ingestion pipeline#7077
waleedlatif1 wants to merge 3 commits into
stagingfrom
codex/knowledge-ingestion-hardening

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Harden connector pagination, hydration, reconciliation, and queue ownership boundaries
  • Bound parser, archive, chunking, embedding, and upstream error handling
  • Add durable quota continuation and explicit skipped and failed processing outcomes
  • Add backward-compatible queue-generation, deferred-processing, and sync-metric schema

Type of Change

  • Bug fix

Testing

  • bun run lint
  • bun run check:audits
  • bun run check:migrations origin/staging
  • bunx turbo run type-check
  • bun run test

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@greptile-apps

greptile-apps Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Too many files changed for review (128 files, 100 file limit).

Bypass the limit by tagging @greptile-apps to review.

@vercel

vercel Bot commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Aug 25, 2026 11:19pm

Request Review

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Aug 25, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 128 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/connectors/google-drive/google-drive-errors.ts
Comment thread apps/sim/lib/chunkers/json-yaml-chunker.ts Outdated
Comment thread apps/sim/lib/chunkers/structured-data-chunker.ts
Comment thread apps/sim/lib/file-parsers/yaml-parser.ts
Comment thread apps/sim/connectors/google-docs/google-docs.ts
Comment thread apps/sim/lib/file-parsers/opendocument-parser.ts
Comment thread apps/sim/lib/knowledge/documents/utils.ts Outdated
@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch from 60ccc13 to ab5457a Compare August 25, 2026 20:50
@waleedlatif1 waleedlatif1 mentioned this pull request Aug 25, 2026
Closed
6 tasks
@waleedlatif1

waleedlatif1 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Aug 25, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 128 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/lib/knowledge/documents/utils.ts
Comment thread apps/sim/lib/chunkers/structured-data-chunker.ts Outdated
Comment thread apps/sim/lib/knowledge/documents/utils.test.ts Outdated
Comment thread apps/sim/lib/chunkers/json-yaml-chunker.ts Outdated
@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch from 94fbb94 to 9c2d9d3 Compare August 25, 2026 22:00
@gitguardian

gitguardian Bot commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@waleedlatif1

waleedlatif1 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch from 9c2d9d3 to 28171cb Compare August 25, 2026 22:08
@waleedlatif1

waleedlatif1 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Aug 25, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 129 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/lib/chunkers/json-yaml-chunker.ts Outdated
cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Aug 25, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 129 files

Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic

Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/lib/chunkers/structured-data-chunker.ts Outdated
Comment thread apps/sim/lib/chunkers/json-yaml-chunker.ts Outdated
Comment thread apps/sim/lib/knowledge/documents/utils.test.ts
@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch from 28171cb to 1320c71 Compare August 25, 2026 22:40
@waleedlatif1

waleedlatif1 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Aug 25, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 129 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/chunkers/json-yaml-chunker.ts Outdated
@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch from 1320c71 to e0a511f Compare August 25, 2026 23:10
@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch from e0a511f to 8e034ed Compare August 25, 2026 23:17
@waleedlatif1

waleedlatif1 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cubic-dev-ai cubic-dev-ai[bot] cubic-dev-ai[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@waleedlatif1