feat(mcp): add include/exclude pattern support to scrape_docs - #477
feat(mcp): add include/exclude pattern support to scrape_docs#477hjhriedel wants to merge 4 commits into
Conversation
arabold
left a comment
There was a problem hiding this comment.
Good changes, thanks for this. My only concern is that this breaks comma in regex pattern. It might be an edge case but there are legitimate use cases for it:
- Regex quantifiers:
/\/v\d{1,3}\// - Glob brace expansion:
**/*.{js,ts} - Literal commas in URLs
The CLI avoids it by accepting repeated --include-pattern / --exclude-pattern flags as arrays. I think the web UI suffers from the same problem as your logic here though.
I think the best option would be to use z.array(z.string()) for MCP. Something like this:
const patternsSchema = z.union([z.string(), z.array(z.string())]).transform((value) =>
typeof value === "string" ? [value] : value,
);| function splitPatterns(patterns: string | undefined): string[] | undefined { | ||
| if (patterns === undefined) return undefined; | ||
| const items = patterns | ||
| .split(",") |
There was a problem hiding this comment.
Could we preserve commas inside a single regex? A pattern that uses a comma, such as a range, would be split into two pieces and stop working. Please split only between patterns, or support escaped commas.
|
Thanks — on top of your union proposal I went one step further, because MCP clients like the opencode TUI only render primitive arguments (arrays get dropped from the tool-call display), so the string form is what actually shows up for users. New behavior:
This keeps the string argument visible in the TUI and safe for regex/glob commas. Updated the schema description and added tests for quantifiers, brace expansion, character classes, escaped commas, and empty segments. |
arabold
left a comment
There was a problem hiding this comment.
Thanks for addressing the comma handling and covering the edge cases with tests.
Thank you for this awesome project! |
Adds
includePatterns/excludePatternssupport to the MCPscrape_docstool.Changes
src/mcp/mcpServer.ts(+32):includePatternsandexcludePatternson thescrape_docstool schema as comma-separated strings (regex patterns wrapped in slashes, e.g./pattern/).splitPatterns()helper splits the comma-separated string into trimmed, non-emptystring[]before passing to the scraper (which keeps receiving arrays as before).The underlying scraper already supported these options; this PR exposes them through the MCP interface.