sponsor Vim development Vim logo Vim Book Ad

fencview.vim : Auto detect CJK and Unicode file encodings.

 script karma  Rating 721/249, Downloaded by 20413  Comments, bugs, improvements  Vim wiki

created by
Ming Bai
 
script type
utility
 
description
Have you ever met that after opening a text file with some kind of encoding you got a bunch of unreadable mess? If so, you need fencview.vim.

Normally, vim detects file encoding based on 'fileencodings' option. For example, if we set this option like this: set fileencodings=ucs-bom,GB2312,big5 Firstly vim try to read file bom. If there is no bom, vim will try to decode the file with GB2312. Unfortunately, the GB2312 and big5 have a lot of overlapped code-points which could lead to a big5-encoded file be decoded with GB2312 by mistake. Then you got mess.

The 'vencview' uses another approach, let me show you how. As we know, take Chinese for example, the occurrence frequency of every character in a normal article is different. For example, '的' is a high frequency character which is encoded to 0xb5c4 in GB2312. However, in Big5 0xb5c4 is the code point for '腔' which is much less frequently used. So we can "guess" the encoding of a file by count up the frequently used characters. It is not 100% accurate but works in most cases.

The user manual is embeded in fencview.vim, refer to it if you'd like to custmize this script.

If you encounter a file that fencview can not handle, you're more than welcome to send it to me in order to improve the algrithm, but take care of the sensitive data before you do that.

How to use
Basically, you need this plugin only when you ran into some messy code. In this case you need fencview to auto detect the file encoding. Either click the menu entry: "Tools->Encoding->Auto Detect" or use this command: :FencAutoDetect and probably that's all.

If it doesn't work, you may try to 'guess' the file encoding by yourself. The menu entry "Tools->Encoding" or the :FencView command will help you do that.

If you need fencview to detect encoding of every file, set g:fencview_autodetect option to 1 in your vimrc. This will slightly impact loading time, so it is disabled by default.

Now FencView allows the use of an external program to determine the file encoding. It may give you better performance, or better accuracy, in some cases. https://github.com/adah1972/tellenc
 
install details
Simply put fencview.vim into 'plugin' directory and that's all.
 

rate this script Life Changing Helpful Unfulfilling 
script versions (upload new version)

Click on the package to download.

package script version date Vim version user release notes
fencview.vim 4.9 2014-03-24 7.0 Ming Bai Fix three issues:
- Non-ASCII file names cause failures after 7.4.122.
- Command ":FencManualEncoding" and menu Tools > Encoding behave differently.
- Modeline does not change the file encoding when a document is reloaded.
fencview.vim 4.8 2012-02-04 7.0 Ming Bai Set and restore cpoptions to make sure line continuation works
fencview.vim 4.7 2011-04-04 7.0 Ming Bai Fix the bug that a modified buffer may be reloaded (:e!) unintentionally.
fencview.vim 4.6 2008-12-11 7.0 Ming Bai Progress bar no longer used for better compatibility; one may use  g:fencview_checklines=0 to check all lines; do not change the current file encoding if no valid multi-byte encoding is detected.
fencview.vim 4.5 2008-09-05 7.0 Ming Bai Add a workaround (UNIX only) for earlier Vim versions where shellescape() is not defined; avoid using `y$' to prevent issues when folding is used (contributed by Ingo Karkat).
fencview.vim 4.4 2008-06-02 7.0 Ming Bai Fix two problems found under Unix:
1) Big5 can be loosely used for CP950;
2) g:legacy_encoding can be wrongly guessed (it should not be used).
fencview.vim 4.3 2007-11-27 7.0 Ming Bai Make it run more smoothly under Windows with my gVim build (http://wyw.dcweb.cn/#download).
fencview.vim 4.2 2007-05-25 7.0 Ming Bai update
fencview.vim 4.1 2007-05-22 7.0 Ming Bai some improvements
fencview.vim 4.0 2007-05-03 7.0 Ming Bai It merges functionality in multienc.vim: allows the use of an external program to determine the file encoding; use the http-equiv meta tag to help decide the file encoding; use the fileencoding modeline to help decide the file encoding; autodetect the encoding of .txt, .htm, and .html files; add a new command FencManualEncoding to override the encoding where autodetection affects the normal ++enc. Also some minor fixes are made.
fencview.vim 3.2 2007-03-04 7.0 Ming Bai bug fix: set syntax on after auto detect
fencview.vim 3.1 2006-12-11 7.0 Ming Bai A lot of improvements.
fencview.vim 2.3 2006-11-28 7.0 Ming Bai Now it can auto detect the file encodings. You just need to select "Auto detect" from the menu or use :FencAutoDetect command to let it detect the fileencoding automatically.
Supported encodings:
utf-8
cp936(GBK)
cp950(big5)
cp932(sjis)
fencview.vim 1.2 2006-11-16 7.0 Ming Bai Some bug fix.
fencview.vim 1.1 2006-11-15 6.0 Ming Bai Now, you can select a file encoding from the menu, just like what you did in Firefox/IE.
Screenshot:
http://mbbill.googlepages.com/fenc.png
fencview.vim 1.0 2006-11-14 7.0 Ming Bai Initial upload
ip used for rating: 3.15.147.53

If you have questions or remarks about this site, visit the vimonline development pages. Please use this site responsibly.
Questions about Vim should go to the maillist. Help Bram help Uganda.
   
Vim at Github