Sitelet https://github.com/NanmiCoder/MediaCrawler/pull/973
Skip to content

fix(storage): map CSV and Excel values to stable headers - #973

Open
saksim wants to merge 1 commit into
NanmiCoder:mainfrom
saksim:codex/upstream-export-schema
Open

saksim wants to merge 1 commit into
NanmiCoder:mainfrom
saksim:codex/upstream-export-schema

Conversation

@saksim

@saksim saksim commented Sep 9, 2026

Copy link
Copy Markdown

Using each record's dictionary insertion order as column order puts values in the wrong columns when records are reordered or have different fields. Reuse the existing file/sheet header and map values by field name.

CSV reads its existing header, leaves missing fields empty, and rejects new fields before appending. Callers can start a new file or explicitly migrate the schema. Excel preserves existing columns and appends newly seen fields on the right, leaving earlier rows empty in those columns. The CSV header read is offloaded with asyncio.to_thread.

Validation: all 7 regression tests fail before the fix. This isolated branch passes all 103 tests (Python 3.11, Windows with PYTHONUTF8=1). Tests reopen actual CSV/XLSX files and cover new writer instances, reordered/missing/unknown fields, and all 5 Excel sheet types.

This branch contains only schema mapping. It does not include the separate formula escaping or atomic-write changes. See docs/export-schema.md. Fork tracking and CI: PR #22. Related report: saksim#6.

@saksim
saksim requested a review from NanmiCoder as a code owner September 9, 2026 03:18
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant