UTF-8 vs Shift-JIS: Why CSV Files Get Garbled

A CSV file that looks fine on one computer can open as a wall of question marks or odd symbols on another, most often in Excel. The file was saved in one encoding and opened in another, and the two disagree about which character each byte stands for.

The fix depends on knowing which encoding the file has and which one the receiving program expects. Most Japanese CSV files involve one of two: UTF-8 or Shift-JIS.

Why does a CSV file look garbled?

A CSV looks garbled when the program that saved it and the program that opens it assume different encodings. A character encoding is the rulebook that maps byte sequences, the numbers a computer stores, to characters, so the same bytes read under another rulebook turn into other characters.

Take a customer name saved on a Japanese Windows system and opened by a tool that expects UTF-8. The bytes are intact, but the tool maps them to the wrong characters, and the name comes out as scrambled symbols. Nothing is lost in the file itself; the reader is applying the wrong rulebook.

The garbage on screen is a clue, not noise. Different mismatches produce different patterns, so the symbols tell you which pairing went wrong. Fixing the file means matching writer and reader, not trying export settings at random.

UTF-8 vs Shift-JIS: how the two encodings differ

UTF-8 can represent every character in every writing system, while Shift-JIS covers Japanese only, so UTF-8 is the broader choice and Shift-JIS the narrower one. In UTF-8, a Japanese kana or kanji typically takes 3 bytes and an ASCII character takes 1, while Shift-JIS stores most Japanese characters in 2 bytes. A file heavy in Japanese text is therefore a little more compact in Shift-JIS, and the table sets the two side by side.

Shift-JIS has a second weakness. Platform-dependent symbols such as circled numbers and some kanji variants cannot be written in standard Shift-JIS. Microsoft's CP932 extends Shift-JIS with the missing symbols, and what Windows labels "Shift-JIS" is almost always CP932.

Two older Japanese encodings, EUC-JP and ISO-2022-JP, still turn up in legacy Unix systems and Japanese email. They rarely appear in a CSV that a business user handles.

What is a UTF-8 BOM, and why does Excel need it?

A UTF-8 BOM is a marker of 3 bytes, EF BB BF, at the very start of a file that tells the reader the file is UTF-8, and Excel on Windows depends on it. BOM stands for Byte Order Mark. Without the marker, Excel falls back to the system default encoding, which is often Windows-1252 or Shift-JIS depending on the machine, and the text turns garbled.

That makes UTF-8 with a BOM the usual answer for a file that will be opened in Excel. A UTF-8 file without a BOM is the plain version, and it is what most programs expect.

The marker has a cost outside Excel. Some Python scripts, command-line tools and web-app CSV importers read the BOM as ordinary data, so an extra invisible character appears in the first column header and a lookup on that header fails. Add the BOM when Excel is the destination and leave it off when a program reads the file.

How to check which encoding a CSV file uses

Check the file's current encoding before converting, because converting from the wrong source encoding damages the text. The symptoms alone are not a safe guide. On Windows, Notepad shows the detected encoding, "UTF-8" or "ANSI", in the status bar at the bottom right, where ANSI means Shift-JIS on a Japanese system.

On Mac or Linux, file -i filename.csv in a terminal reports the detected character set. The sakuttoAI CSV Viewer also detects the encoding automatically when a file is uploaded.

Editors can mislead. A text editor may show "UTF-8" for a file that is not UTF-8, so if the text still looks broken under that setting, try the other encoding.

How to convert a CSV to UTF-8 or Shift-JIS

To get a UTF-8 file out of Excel, open the CSV, choose File → Save As, and pick "CSV UTF-8" from the format list. Excel writes UTF-8 with a BOM, and Excel can reopen that file correctly. This route needs no extra software, though it can fail on a file that mixes several encodings.

For a file Excel cannot open cleanly, or when the target is Shift-JIS, a browser tool is the shorter path. sakuttoAI's CSV Encoding Converter switches a file between UTF-8 and Shift-JIS, detects the source encoding, and runs in the browser, so the file is never uploaded to a server. A garbled customer export from an old system can be converted to UTF-8 in one pass this way.

Direction matters for data loss. Shift-JIS to UTF-8 loses nothing, because UTF-8 covers every Shift-JIS character. UTF-8 to Shift-JIS can replace characters Shift-JIS lacks, such as emoji and some kanji, with "?".

Which to choose: UTF-8 or Shift-JIS for your CSV

Choose UTF-8 unless the destination demands Shift-JIS. The destination decides, so match it to the case:

• Opened directly in Excel on Japanese Windows: UTF-8 with a BOM, or Shift-JIS when no BOM-aware step is available.
• Read by a program, script or web service: UTF-8 without a BOM.
• Mixed scripts such as Chinese, Korean or emoji: UTF-8, since Shift-JIS cannot hold them.
• A legacy business system or a trading partner that specifies Shift-JIS: Shift-JIS, as the specification requires.

A new CSV with no constraint pulling it elsewhere belongs in UTF-8. It has the widest character coverage and the widest tool support, and converting to Shift-JIS later costs one step.

Mistakes that corrupt a CSV during conversion

The worst mistakes are saving over a file that already shows garbled text and converting from the wrong source encoding. Both make the damage permanent.

• Saving over a broken file: if a file already shows garbled text and is saved with "Save with Encoding → UTF-8", the broken characters are written out as UTF-8 and the corruption cannot be undone. Reopen the file with the right source encoding first.
• Converting from the wrong source: a Shift-JIS file treated as something else produces new garbage on top of the old. Identify the real encoding, then convert once.
• Going from UTF-8 to Shift-JIS blindly: unsupported characters become "?" and stay that way. Check the data for emoji, circled numbers and rare kanji before converting.
• Overwriting the original: save the converted copy under a new name and keep the source file until the result opens correctly.

Summary: match the encoding to where the CSV goes

Garbled CSV text is an encoding mismatch between the program that saved the file and the one that opens it. Check the file's current encoding first, then save or convert it to what the destination expects: UTF-8 with a BOM for Excel, plain UTF-8 for programs, and Shift-JIS only when a system or partner requires it.

Convert once, from the real source encoding, and keep the original. The CSV Encoding Converter handles the switch in the browser when Excel's own Save As is not enough.