How to Fix CSV Encoding Issues: UTF-8, BOM, and Windows-1252

An author's name reaches a CSV importer as bytes. The decoder decides which characters those bytes become.

Böll, Heinrich      what the shop typed
Böll, Heinrich     UTF-8 bytes decoded as Windows-1252
B�ll, Heinrich      Windows-1252 bytes decoded as UTF-8

The first broken value is mojibake. Its bytes still hold the original information, and the importer read them with the wrong character encoding. The second holds �, the Unicode replacement character, which a UTF-8 decoder writes when the input carries a byte sequence it cannot decode. Once an earlier conversion has replaced a character with ? or � and saved the result, the original character is gone.

CSV itself requires no one encoding. RFC 4180 registers charset as an optional parameter of the text/csv media type, so a sender can declare an encoding while the file travels with MIME metadata. A file picked from a desktop reaches the importer as bytes with no such parameter. A byte order mark can identify some Unicode files, UTF-8 among them, and the Unicode consortium states the rest in one line. "If there is no BOM, the encoding could be anything."

The file in this guide makes that harder. It holds 26 rows from an antiquarian bookshop, assembled from two exports that were concatenated without transcoding either of them. The newer system wrote UTF-8. The old till wrote Windows-1252. The encoding therefore changes partway down the byte stream.

1Cat. no.,Author,Title,Lang,Price (EUR),Shelf2CB-0208,"Достоевский, Фёдор",Записки из подполья,ru,€27.00,D13CB-0211,"Ōe, Kenzaburō",The Silent Cry,en,€33.00,C4⋮2 rows not shown6CB-0223,"Fauré, Gabriel",Lettres à ses éditeurs,fr,€45.00,B4⋮2 rows not shown9CB-0236,José Ãlvarez,Cartas de invierno,es,€13.40,B1⋮6 rows not shown16CB-0104,"Böll, Heinrich",Ansichten eines Clowns,de,€18.50,A3⋮3 rows not shown20CB-0121,"O’Brien, Flann",The Third Policeman,en,€19.00,C2⋮1 rows not shown22CB-0130,"?e, Kenzabur?",A Personal Matter,en,€31.00,C4⋮4 rows not shown27CB-0151,"Undset, Sigrid",Kransen,no,€17.30,D1

The rows fail in different ways. Row 2 needs UTF-8 for Cyrillic and row 3 needs it for two macrons. Row 6 writes é as e plus a combining accent, which is valid Unicode and changes how the value compares. Row 9 already holds mojibake from an earlier conversion. Rows 16 to 27 came off the Windows-1252 export, including the curly apostrophe in row 20. Row 22 is worse. Its two macrons became question marks before the file reached the importer, so no decoder recovers them.

A BOM is an encoding signature inside the file

A byte order mark sits at the front of the byte stream itself, which is the one place a media type never reaches. Unicode defines it as "a signature defining the byte order and encoding form" and gives it a sequence for each Unicode encoding form.

EF BB BF       UTF-8
FF FE          UTF-16 little-endian
FE FF          UTF-16 big-endian
FF FE 00 00    UTF-32 little-endian
00 00 FE FF    UTF-32 big-endian

The two little-endian marks overlap, since FF FE opens both of them, so a reader has to test the four-byte sequence before the two-byte one or it reads every UTF-32 file as UTF-16. UTF-8 has no byte order to mark, because a decoder reads it one byte at a time. Its mark carries the signature and nothing else.

Excel makes the effect visible. Microsoft's API lists the CSV save formats as xlCSV, xlCSVWindows, xlCSVMac, xlCSVMSDOS and xlCSVUTF8, described as "CSV", "Windows CSV", "Macintosh CSV", "MSDOS CSV" and "UTF8 CSV". Its support page says a UTF-8 CSV opens normally in Excel when the file was saved with a BOM. Without that signature the normal open path can read the same bytes as something else.

NetSuite documents that failure in its own exports. It writes CSV as UTF-8 with no BOM, and its documentation says Excel then uses a different encoding, so non-ASCII characters may not show correctly. Its recommendation is to export to Excel first and save the workbook back to CSV, because Excel adds a BOM to what it writes.

The CSV data did not change between those two paths. The extra bytes at the front changed what the reader knew before decoding the first field.

A few bytes can change the detector's answer

Updog Importer resolves an unmarked file in a fixed order, and the order decides how much of the answer is a guess.

The importer checks the opening bytes for a BOM first. Without one, it tries a strict UTF-8 decode of the first 64KB, and a sample that survives that decode settles the encoding with no statistics involved. Only a sample that fails goes to chardet, which returns the encoding with the highest confidence for those bytes.

A UTF-8 verdict then gets one more check, against the whole file, and it makes no difference whether the verdict came from the mark or from the sample. A sample can be ASCII-clean while legacy bytes sit deeper in, so the importer decodes every byte in strict mode before it accepts the verdict. When that decode succeeds, its output becomes the file text. When it fails, the importer re-detects over a window anchored at the first non-ASCII byte. A byte order mark is a hypothesis here like any other, and a file that fails the strict decode loses it.

The strict pass reads the whole file, so it is not free. What it is not is a second decode thrown away. Its output is the text the importer keeps.

When chardet has no answer at all, the fallback is Windows-1252. In the web encoding model its decoder maps every byte value to a code point, so no byte sequence can make that fallback fail.

Common CSV import errors covers the rest of the opening pass, and how to import a CSV into MySQL shows the same chain inside a complete import flow.

The statistical step carries less certainty than the rest. These results come from chardet 2.2.0, the version Updog Importer ships, called directly rather than through the chain above. Each file holds the header shown, then its rows, then a trailing newline, encoded to Windows-1251 or Windows-1252 before detection. The Cyrillic rows in order are Иван Петров, Анна Смирнова and Ольга Кузнецова. The Western European rows are René;Zürich, Hervé;Genève, Noël;Fribourg, Chloé;Nyon and Amélie;Sion.

More bytes did not steadily improve the answer. One Windows-1251 row was classified correctly. A second row moved the verdict to GB18030. A third moved it back. Changing the header from name to id;name, two ASCII characters, moved the one-row file from windows-1251 onto a Latin table.

The Windows-1252 sample shows a different case. chardet calls it iso-8859-1 and the browser still produces the intended text. Under the WHATWG Encoding Standard the labels latin1, iso-8859-1, cp819 and ascii all select the Windows-1252 decoder. "These are synonyms: latin1 and ascii are just labels for windows-1252." A wrong label reached the right decoder.

The Windows-1251 samples have no equivalent alias. When the detector answers GB18030 or iso-8859-1, the same bytes go through a different table and become different characters.

That is the limit of statistical detection. It returns the encoding that best explains the bytes it sampled, and a few more bytes can change that explanation. Treat the result as a hypothesis. For UTF-8, a strict decode proves the byte sequence is valid UTF-8. For a legacy encoding, the importer has no equivalent proof.

The corruption pattern points to the decoder

Two failures leave different text behind, and each one starts from different bytes.

é    UTF-8 bytes decoded as Windows-1252
�     Windows-1252 bytes decoded as UTF-8

Take é. UTF-8 stores it as the two bytes C3 A9. Windows-1252 assigns a character to each byte on its own, so C3 becomes Ã and A9 becomes ©. The decoder accepts both bytes and returns valid Unicode text. The result is readable, reversible mojibake.

Windows-1252 stores the same é as the single byte E9, which cannot stand alone in UTF-8. A UTF-8 decoder running in replacement mode substitutes U+FFFD, the replacement character, for that malformed input.

The two failures preserve different amounts of information. In José the characters still map back to the original bytes once you know Windows-1252 was the wrong decoder. Re-encode é as Windows-1252 and the bytes are C3 A9 again. Decode those as UTF-8 and é returns.

Jos� is different. U+FFFD records that decoding failed, and it records nothing about which byte was there. An application that saves the decoded string and drops the original bytes cannot reconstruct the missing E9 from �.

That difference is what the strict decode in the previous section is for. During detection Updog Importer never accepts malformed UTF-8 and papers over it with �. The strict decoder rejects the UTF-8 hypothesis instead, which leaves the bytes available for another encoding. The decode that finally produces the text runs in replacement mode, so a file whose UTF-8 verdict survives the strict check can still come back carrying replacement characters. The next section is that file.

One encoding decision covers the whole file

A CSV parser applies one character encoding to the whole byte stream. This file does not have one.

The mark at its front says UTF-8, and that is where the importer starts. The strict decode of the whole file then fails, because the twelve Windows-1252 rows hold byte sequences no UTF-8 decoder accepts. A byte order mark is a hypothesis in this chain, and this file is where the hypothesis loses. What decides the outcome after that is chardet, running on the same bytes.

When it answers UTF-8, the fourteen UTF-8 rows decode correctly and the twelve Windows-1252 rows do not.

CB-0104,"B�ll, Heinrich",Ansichten eines Clowns,de,�18.50,A3
CB-0121,"O�Brien, Flann",The Third Policeman,en,�19.00,C2

When it answers Windows-1252, the opposite half loses. Those twelve rows survive intact, and the UTF-8 sequences above them break apart, so C3 A9 reads as é.

The balance moves with a small change in the byte stream. Holding the twelve Windows-1252 rows fixed and varying only the number of UTF-8 rows above them, through the full decode chain on chardet 2.2.0 under Node 24.19.0, the file crosses over between the eighth row and the ninth.

Fifty-six bytes of new rows moved the loss from one half of the file to the other. The right-hand column is the same experiment with the three mark bytes deleted, and it never crosses over. That is worth reading twice. The mark's value as a signature was already rejected by the strict decode, so what those three bytes do here is act as three more bytes of evidence in a statistical count.

On a file that is genuinely UTF-8 the mark behaves the way it is meant to. The strict decode confirms it, detection never runs, and no distribution of bytes can overrule it. The mark loses its authority only on a file that is not what it claims to be.

Detecting each row separately does not fix this. A single row gives the detector far less evidence than the whole file, and the earlier table already showed short samples moving between windows-1251, GB18030 and iso-8859-1 after a two-character change. Updog Importer therefore keeps one encoding decision for the file. Where different parts of a file were written with different encodings, detection can identify the encoding that best explains the bytes as a whole. It cannot make both halves correct at once.

A correct decode cannot recover earlier data loss

A decoder can reinterpret the bytes that remain. It cannot reconstruct characters that an earlier step already replaced.

Row 22 of the bookshop file contains ?e, Kenzabur?. Windows-1252 has no byte for Ō, which is U+014C, so the old till substituted a question mark before the current CSV existed. The byte in the file is 3F, an ordinary ASCII question mark, and nothing in that byte says whether the shop typed it or an encoder put it there. The macron is not in the file to be found.

Row 9 fails differently. It contains José Ãlvarez, and those characters are valid UTF-8, so the detector chooses UTF-8 and chooses correctly. The decoded text is still wrong, because the mojibake was made earlier and then saved as text.

original text
José

UTF-8 bytes decoded as Windows-1252
José

that damaged string saved as UTF-8
José

The current decoder sees only the last step. Nothing available to it says that é was once é.

The first case keeps enough information for a mechanical reversal. Re-encode José through Windows-1252 and the bytes are 4A 6F 73 C3 A9 again, which read as José. That works here because Windows-1252 assigns a character to all 256 byte values, so nothing was dropped on the way through it.

The other two are lossy. � records that decoding failed and keeps no copy of the bytes it replaced, and one � does not always stand for one lost byte. ? says even less, because it is also a character a person could have typed.

Encoding detection is finished at that point. Recovering Ōe or deciding that José means José is a correction to the data, and that decision belongs to the person who knows what the value was meant to say.

NFC gives equivalent text one representation

Decoding produces Unicode text, and the same visible value can still have more than one representation. Updog Importer normalises every cell after the decode. It removes a defined set of zero-width characters, replaces non-breaking spaces with ordinary spaces, normalises the value to NFC, and trims the ends.

Row 6 shows why NFC matters. In this file Fauré arrives as six code points, because the final é is stored as e followed by U+0301, the combining acute accent.

Fauré    F a u r e ◌́     6 code points
Fauré     F a u r é        5 code points

The two strings render the same and are canonically equivalent in Unicode. They are not the same sequence of code points, so binary or code-point equality tells them apart while a collation-aware comparison can treat them as equal. NFC converts both representations to the same form.

That matters when a value takes part in identity. Where an application matches imported rows on a name, a reference or any other text field, two visually identical values should not become different keys because one source used a combining character.

Updog Importer hands over normalised text. Your application and your database still decide which fields identify a row and how equality is evaluated. NFC removes one source of accidental difference before that decision is made.

A misdecoded BOM becomes ordinary text

U+FEFF sits in the set of zero-width characters that cleanup removes from a cell, which handles a mark that survives into the middle of a merged file.

A mark at byte zero takes a different path, since the importer reads it before any cell exists. Three bytes of EF BB BF make UTF-8 the first candidate.

The rest of the file can contradict that candidate. Put Windows-1252 data behind those three bytes and the strict decode fails on the body, so the importer rejects UTF-8 and carries on through detection. When detection settles on Windows-1252, that decoder has no concept of a signature. It reads the three bytes as three characters.

EF BB BF    a UTF-8 byte order mark

decoded as Windows-1252

ï » ¿

A header of Author therefore reaches column matching as Author. Cleanup cannot help, because U+FEFF is no longer in the string. The same cleanup turns \uFEFFAuthor back into Author and leaves Author exactly as it is.

The bytes began as encoding metadata, and the wrong decoder turned them into data.

How to build CSV column mapping in React follows the next boundary and shows what happens when a header reaches the matcher under a name the schema does not hold.

A successful decode can still produce the wrong text

The importer always reaches a decoded string. A UTF-8 candidate can fail its strict check inside the chain, and the fallback path then continues until a decoder accepts the bytes. Windows-1252 is the last of them, so an import never fails because the encoding could not be decoded.

Encoding damage therefore arrives in the grid as ordinary text.

The bookshop file reports 26 rows, 6 of 6 columns matched, and 0 rows with validation errors. Twelve rows hold a replacement character after the decode, and nine hold an author name different from the one the shop wrote.

CB-0231   Sebald, W. G.              Die Ausgewanderten
CB-0236   José Ãlvarez              Cartas de invierno
CB-0104   B�ll, Heinrich             Ansichten eines Clowns
CB-0107   M�rquez, Gabriel Garc�a    El coronel no tiene quien le escriba

All four rows are valid CSV. All four author values are valid strings. The importer can check the shape of a row and still have no basis for deciding whether B�ll or José is the name the shop intended.

As of August 2026, Updog Importer does not show the chosen encoding and takes no prop for overriding it. The decode runs in a worker before header detection starts. That worker returns the decoded text, the detected delimiter and the encoding verdict, and the CSV parser takes the text and the delimiter. The verdict stops at that boundary, so nothing downstream can tell the person which decoder produced what they are reading.

The values become the only visible clue. A run of é, ü or ’ points towards UTF-8 bytes read through a single-byte decoder. A run of � points towards malformed input that went through a replacement-mode decode somewhere in the file's history.

Those patterns can tell a person that the text deserves attention. They cannot prove what the original value was.

Text formats do not all leave the same encoding question

The chain in this article exists because the importer receives an unlabelled byte stream and still has to turn it into text. A CSV commonly arrives that way. The other formats do not, and Updog Importer currently runs the same decoding stage ahead of its CSV, JSON and XML parsers even though the formats define encoding differently.

JSON has the narrowest contract of the three. RFC 8259 states that "JSON text exchanged between systems that are not part of a closed ecosystem MUST be encoded using UTF-8", and that implementations "MUST NOT add a byte order mark (U+FEFF) to the beginning of a networked-transmitted JSON text". A JSON reader can treat UTF-8 as the format rule rather than infer a legacy encoding from the bytes.

XML carries more information of its own. A document can declare its encoding, every processor "MUST be able to read entities in both the UTF-8 and UTF-16 encodings", and entities in UTF-16 "MUST" begin with a byte order mark while UTF-8 entities "MAY". An entity with no declaration and no external information has to be legal UTF-8 or UTF-16 or the processor reports a fatal error. The spec's appendix on recognising an encoding from the opening bytes is non-normative, so it describes the technique rather than requiring it.

An .xlsx takes a different path again. It is an Open XML package rather than one text stream, so the importer opens the package and parses its workbook, worksheet and shared-string parts under the XML rules above. No encoding detector runs over the workbook as a whole.

That leaves the unlabelled desktop CSV as the case this article is really about. A file can hold text with no in-band declaration, no byte order mark, and no transport metadata left beside it. The importer has bytes and has to choose how to read them.

A team that controls its own exports can remove the ambiguity. A UTF-8 CSV with a byte order mark hands the reader an in-band signature that survives the strict decode. An .xlsx keeps its text inside a package whose parts carry their own encoding rules.

A team receiving whatever an older till, ERP or desktop application wrote does not control that boundary, and detection can only choose the encoding that best explains the bytes it was given.

The order is part of the encoding policy

A detector does not settle the encoding on its own. It supplies a candidate, and the importer decides which evidence outranks which.

The pass in this article uses a fixed order. A byte order mark gets the first vote. Without one, the detector proposes a candidate from the bytes it samples. A UTF-8 candidate then has to survive a strict decode of the whole file. When the browser cannot decode the candidate at all, or the detector produces no usable answer, Windows-1252 closes the chain.

import { detect } from "chardet";

export async function decodeCsvFile(file: File): Promise<string> {
  const bytes = new Uint8Array(await file.arrayBuffer());

  const hasUtf8Bom =
    bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf;

  if (hasUtf8Bom) {
    try {
      return new TextDecoder("utf-8", { fatal: true }).decode(bytes);
    } catch {
      // The signature and the body disagree. Continue with detection.
    }
  }

  const label = detect(bytes);

  if (label?.toLowerCase() === "utf-8") {
    try {
      return new TextDecoder("utf-8", { fatal: true }).decode(bytes);
    } catch {
      // UTF-8 was only a statistical candidate.
    }
  }

  if (label) {
    try {
      return new TextDecoder(label).decode(bytes);
    } catch {
      // The detector can name an encoding the browser cannot decode.
    }
  }

  return new TextDecoder("windows-1252").decode(bytes);
}

fatal: true turns UTF-8 decoding into a validity check. Without it, malformed sequences become U+FFFD and the decoder returns a string anyway. With it, a malformed sequence rejects the candidate and the original bytes stay available for another decoder.

That check proves less than it looks. It proves the bytes form a valid UTF-8 sequence, and not that the source wrote UTF-8. UTF-16LE holding ASCII is the clearest case, since 69 00 64 00 is valid UTF-8 that decodes to i, a null, d, a null. A strict decode accepts it. That is why the mark votes first, and why a UTF-16 file with no mark is the case this pass does not settle.

The second try covers a different boundary. chardet and the browser do not support the same set of encodings. chardet 2.2.0 can return UTF-32LE or UTF-32BE, the WHATWG Encoding API defines no UTF-32 decoder, and new TextDecoder(label) throws a RangeError on a label it does not know. A detector can recognise an encoding the browser refuses to build.

Windows-1252 closes the chain because its web decoder maps the full byte range. The importer can always produce text, and producing text does not prove the text is correct.

With Updog Importer this pass runs before the schema sees the file. The application defines the six bookshop fields.

import type { DataEditorColumn } from "@updog/data-editor";

export const columns: DataEditorColumn[] = [
  {
    id: "catalogRef",
    title: "Cat. no.",
    validators: [{ type: "required" }, { type: "unique" }],
  },
  { id: "author", title: "Author", validators: [{ type: "required" }] },
  { id: "title", title: "Title", validators: [{ type: "required" }] },
  { id: "language", title: "Lang" },
  { id: "priceEur", title: "Price (EUR)" },
  { id: "shelf", title: "Shelf" },
];

The schema validates those six fields after the decode. It cannot tell from the string alone whether Böll, Böll or B�ll is the author's intended name.

Those three forms bring the article back to where it started. Böll is the value the shop meant to keep. Böll and B�ll expose different losses and reinterpretations somewhere in the file's encoding history.

The importer can make its decoding policy deterministic. It cannot ask the bytes what the person meant. Once decoding has exhausted what the bytes can prove, correcting the value belongs to the person who knows the data.