Data processes / 05
Automate CSV imports. Handle errors precisely.
The file arrives every day. Someone corrects columns, uploads it and then checks error messages. DataSource turns this recurring task into a documented import with clear rules for incomplete and invalid data.
The starting point
The format is simple. The data rules often are not.
A supplier provides products using semicolon separators and German decimal commas. Product numbers contain leading zeros, and descriptions may contain line breaks. Treating the file as plain text lines damages this data.
We review the actual file contract: encoding, quoting, column names, required values, dates and what constitutes a complete file. SFTP or a protected upload can provide the input.
What DataSource delivers
File receipt. Preparation. Import.
File receipt
Reliably detect completed files, for example through an agreed atomic rename or accompanying marker. Do not import a transfer still in progress.
Preparation
Read with a CSV parser, normalize fields according to their types and validate business keys. Report errors clearly with file and line or record references.
Import
Implement staging, preview and target imports to suit the system. File hashes, import IDs and acknowledged records prevent unintentional duplicate imports.
Technical scenario / Illustrative example
Preserve leading zeros and normalize decimal values.
Illustrative mapping of a valid row. The product number remains a string. Invalid prices are not silently set to zero; they are treated as correctable errors.
Example data and proposed rules. The actual interfaces and business rules are assessed for your project.
artikel;preis;land
004218;49,90;at
→ {
"sku": "004218",
"price": "49.90",
"countryCode": "AT"
}Error handling, security and operations
Partial imports: validate and operate.
Partial imports
Decide in advance whether an invalid row stops the entire import. Dependent records may require joint approval.
Retry
Distinguish file versions from record keys. A corrected file can be processed without triggering already imported operations again.
Operations
Monitor expected delivery times, missing files and changed columns. Do not include credentials or sensitive row content in general error reports.
Technical questions
Details that matter before implementation.
CSV · SFTP · Encoding · Mapping · Validation · Import
Can an Excel file be processed in the same way?
XLSX is a different format with its own type and formula rules. We can support it as a separate input; it is not simply renamed as CSV.
What happens when new columns appear?
The file contract defines whether additional columns are tolerated. Missing required columns or changed meanings trigger a targeted review.
Do valid rows always need to wait?
That depends on business consistency requirements. A partial import may be suitable for independent products, but not necessarily for a single related order.
For technical clarification
Start with your actual workflow.
Provide an anonymized original file, the current import guide, target system and typical error messages. Examples with accented characters, null values and leading zeros are especially useful.
In our initial discussion, we clarify available access, data ownership and a narrowly scoped first data flow. We then agree on scope, responsibilities and commercial terms.
Anonymized examples are enough to get started. Please do not send passwords, tokens or personal data from production systems in your initial inquiry.
Related challenges
Your next connection in context.
JSON / XML data transformation
The data is technically readable but still rejected. Field names, types or structures do not match the target application. We develop traceable mappings that also handle empty values, edge cases and changes to the source format.
Automate product data workflows
The ERP supplies master data, the PIM supplies text and a supplier provides technical attributes. Manual lists connect this information only temporarily. We create a defined data flow from each authoritative field to a validated channel export.
Connect SQL databases
Reporting, portals or business applications need data from another database. We plan a connection that accounts for access rights, consistent data states and the rules of the applications involved.