[Ubuntu] Remove Byte Order Mark (BOM) from files recursively [Forward article]

Original article: http://www.yiiframework.com/wiki/570/remove-byte-order-mark-bom-from-files-recursively/

 

The problem was that all my web applications ran normally on localhost, but on server the Greek characters (or any other no-english characters) displayed with problems.

So, I needed to remove BOM from hundreds view files of Yii from a lot of Yii projects manually. Netbeans has an option to keep files Encoding in utf-8 but not utf-8 without BOM.

My previous solution was the converting in utf-8 without BOM encoding one by one file on notepad++ consuming a lot of my time!

Many servers has not this issue but for other servers this is important. So, after of two hours searching I found a fast way to do that by commands.

(If you have windows, install cygwin first)

1) Open a shell command, go into your root folder that contains the project

2) Run this command

grep -rl $'\xEF\xBB\xBF'  /home/rootfolder/Yii_project  > file_with_boms.txt

3) Now, Run this one

while read l; do sed -i '1 s/^\xef\xbb\xbf//'  $l; done < file_with_boms.txt

Thats it! The BOM will be removed from all files that contained it. Now you can upload your project on your server.

Note: Because I didn't use this way many times and I don't know if it works properly for all cases and files, make first a backup of your project! :)

 

 

shell script for replacing tab with spaces and remove bom head from file.

find /path/to/your/folder -type f -exec sed -i 's/\t/    /g' {} \; && grep -rl $'\xEF\xBB\xBF'  /path/to/your/folder  > file_with_boms.txt && while read l; do sed -i '1 s/^\xef\xbb\xbf//'  $l; done < file_with_boms.txt && rm file_with_boms.txt