Multi-Format CSV Parser with Deduplication When ingesting brokerage data across multiple files or format versions, first map all edge cases (header variations, footer disclaimers, BOM markers). Auto-detect format from header row. Strip footers and BOM before parsing. Build a deduplication key (e.g., date + ticker + quantity + price) to merge overlapping transaction rows across files. Validate final dataset against user-stated holdings to catch parsing errors early. ---