[MOD] Generate readable non-english SEF titles (sNews 1.6, 1.5)

[codetwist]

If Your site is in english You don't need even look into this For others it might be of some interest.

In this thread (look in posts) following languages are covered:
 - Latvian
 - Turkish (by courtesy of selcukhoca)
 - Greek (by courtesy of reanimator)
 - Swedish (by courtesy of agentsmith)
 - Spanish and Catalan (by courtesy of bramsyuur, rubenpol)
 - French (by courtesy of Sven)
 - German (by courtesy of reallySimple)
 - Slovak and Czech (by courtesy of oniip)
 - Polish (by courtesy of plaszcz)
 - Portuguese (by courtesy of Dener)
 - Russian (by courtesy of Eugeny Kem)
 - Croatian (by courtesy of Luka)
 - Romanian (by courtesy of Vasile Rusnac)

When I was creating articles in local language - generated urls were messy - as all language specific characters were simply thrown away. For example, if article is called 'Grāvī grūti' the default sef will look like 'grv-grti' - not so easy to read and use. And there is need to keep those sef unique as well.

After applying this mod on sNews javascripts, for same 'Grāvī grūti' title following sef will be built by default : 'gravi-gruti'. While not exactly original title - this is still readable and easier to use.

This mod requires following changes in javascript function genSEF() located at js() (end of snews.php):

Code:

// generate SEF urls
function genSEF(from,to) {
if (allowsef == true) {
// MOD - 20070106 - Replaces characters specific to local language with available lowercase equivalent
// Original code
// var str = str.toLowerCase();
var str = deLocalize(from.value);
str = str.toLowerCase();
// End of MOD - 20070106 - Replaces characters specific to local language with available lowercase equivalent
//str = str.replace(/[^a-zA-Z 0-9]+/g,'');
str = str.replace(/[^a-z 0-9]+/g,'');
str = str.replace(/\s+/g, "-");
to.value = str;
}
}

and introduction of new javascript function deLocalize() right after genSEF (example here is for latvian):

Code:

    // MOD - 20070106 - New function to replace characters specific to local language with available lowercase equivalent
// Deal with local characters (latvian)
function deLocalize( inStr ) {
var outStr = inStr;
outStr = outStr.replace(/[āĀ]/g, 'a');
outStr = outStr.replace(/[čČ]/g, 'c');
outStr = outStr.replace(/[ēĒ]/g, 'e');
outStr = outStr.replace(/[ģĢ]/g, 'g');
outStr = outStr.replace(/[īĪ]/g, 'i');
outStr = outStr.replace(/[ķĶ]/g, 'k');
outStr = outStr.replace(/[ļĻ]/g, 'l');
outStr = outStr.replace(/[ņŅ]/g, 'n');
outStr = outStr.replace(/[šŠ]/g, 's');
outStr = outStr.replace(/[ūŪ]/g, 'u');
outStr = outStr.replace(/[žŽ]/g, 'z');
return outStr;
}
// End of MOD - 20070106 - New function to replace characters specific to local language with available lowercase equivalent

After applying this generated urls were at least readable if a little misspelled   

Rights
What is my work in this that is released to public domain.

Modified script is used in administration part of sNews - so no public access demo for now.

For other languages deLocalize() should be tweaked accordingly Summary of them for other languages can be found in next post.

May be those, who find this mod handy can post their deLocalize()  versions here, so others can reuse them as well

[codetwist]

This post contains copies of all contributed character transliterations from this thread as well as previous one. Origins of older deLocalize() versions summarized in this post can be found here

Turkish:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[çÇ]/g, 'c');
    outStr = outStr.replace(/[ğĞ]/g, 'g');
    outStr = outStr.replace(/[ıI]/g, 'i');
    outStr = outStr.replace(/[öÖ]/g, 'o');
    outStr = outStr.replace(/[şŞ]/g, 's');
    outStr = outStr.replace(/[üÜ]/g, 'u');
    return outStr;
}

Greek:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[άΆαΑ]/g, 'a');
    outStr = outStr.replace(/[βΒ]/g, 'b');
    outStr = outStr.replace(/[γΓ]/g, 'g');
    outStr = outStr.replace(/[δΔ]/g, 'd');
    outStr = outStr.replace(/[έΈεΕ]/g, 'e');
    outStr = outStr.replace(/[ζΖ]/g, 'z');
    outStr = outStr.replace(/[ήΉηΗ]/g, 'i');
    outStr = outStr.replace(/[θΘ]/g, 'th');
    outStr = outStr.replace(/[ίΊιΙ]/g, 'i');
    outStr = outStr.replace(/[κΚ]/g, 'k');
    outStr = outStr.replace(/[λΛ]/g, 'l');
    outStr = outStr.replace(/[μΜ]/g, 'm');
    outStr = outStr.replace(/[νΝ]/g, 'n');
    outStr = outStr.replace(/[ξΞ]/g, 'x');
    outStr = outStr.replace(/[όΌοΟ]/g, 'o');
    outStr = outStr.replace(/[πΠ]/g, 'p');
    outStr = outStr.replace(/[ρΡ]/g, 'r');
    outStr = outStr.replace(/[σΣ]/g, 's');
    outStr = outStr.replace(/[τΤ]/g, 't');
    outStr = outStr.replace(/[ύΎυΥ]/g, 'y');
    outStr = outStr.replace(/[φΦ]/g, 'f');
    outStr = outStr.replace(/[χΧ]/g, 'ch');
    outStr = outStr.replace(/[ψΨ]/g, 'ps');
    outStr = outStr.replace(/[ώΏωΩ]/g, 'o');
    return outStr;
}

Swedish:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[åÅ]/g, 'aa');
    outStr = outStr.replace(/[äÄ]/g, 'ae');
    outStr = outStr.replace(/[öÖ]/g, 'oe');
    return outStr;
}

Spanish and Catalan:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[ñÑ]/g, 'n');
    outStr = outStr.replace(/[çÇ]/g, 'c');
    outStr = outStr.replace(/[üÜ]/g, 'u');
    outStr = outStr.replace(/[ïÏ]/g, 'i');
    outStr = outStr.replace(/[áÁ]/g, 'a');
    outStr = outStr.replace(/[àÀ]/g, 'a');
    outStr = outStr.replace(/[éÉ]/g, 'e');
    outStr = outStr.replace(/[èÈ]/g, 'e');
    outStr = outStr.replace(/[íÍ]/g, 'i');
    outStr = outStr.replace(/[óÓ]/g, 'o');
    outStr = outStr.replace(/[òÒ]/g, 'o');
    outStr = outStr.replace(/[úÚ]/g, 'u');
    return outStr;
}

French:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[àÀ]/g, 'a');
    outStr = outStr.replace(/[âÂ]/g, 'a');
    outStr = outStr.replace(/[éÉ]/g, 'e');
    outStr = outStr.replace(/[èÈ]/g, 'e');
    outStr = outStr.replace(/[êÊ]/g, 'e');
    outStr = outStr.replace(/[îÎ]/g, 'i');
    outStr = outStr.replace(/[ôÔ]/g, 'o');
    outStr = outStr.replace(/[ùÙ]/g, 'u');
    outStr = outStr.replace(/[üÜ]/g, 'u');
    return outStr;
}

German:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[Ää]/g, 'ae');
    outStr = outStr.replace(/[Öö]/g, 'oe');
    outStr = outStr.replace(/[Üü]/g, 'ue');
    outStr = outStr.replace(/[ß]/g, 'ss');
    return outStr;
}

Slovak and Czech:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[áÁ]/g, 'a');
    outStr = outStr.replace(/[éÉ]/g, 'e');
    outStr = outStr.replace(/[íÍ]/g, 'i');
    outStr = outStr.replace(/[ýÝ]/g, 'y');
    outStr = outStr.replace(/[óÓ]/g, 'o');
    outStr = outStr.replace(/[úÚ]/g, 'u');
    outStr = outStr.replace(/[ôô]/g, 'o');
    outStr = outStr.replace(/[ää]/g, 'a');
    outStr = outStr.replace(/[ůů]/g, 'u');
    outStr = outStr.replace(/[ěĚ]/g, 'e');
    outStr = outStr.replace(/[šŠ]/g, 's');
    outStr = outStr.replace(/[čČ]/g, 'c');
    outStr = outStr.replace(/[řŘ]/g, 'r');
    outStr = outStr.replace(/[ťŤ]/g, 't');
    outStr = outStr.replace(/[žŽ]/g, 'z');
    outStr = outStr.replace(/[ľĽ]/g, 'l');
    outStr = outStr.replace(/[ĺĹ]/g, 'l');
    outStr = outStr.replace(/[ďĎ]/g, 'd');
    outStr = outStr.replace(/[ňŇ]/g, 'n');
    return outStr;
}

Polish:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[ąĄ]/g, 'a');
    outStr = outStr.replace(/[ćĆ]/g, 'c');
    outStr = outStr.replace(/[ęĘ]/g, 'e');
    outStr = outStr.replace(/[łŁ]/g, 'l');
    outStr = outStr.replace(/[ńŃ]/g, 'n');
    outStr = outStr.replace(/[óÓ]/g, 'o');
    outStr = outStr.replace(/[śŚ]/g, 's');
    outStr = outStr.replace(/[żŻ]/g, 'z');
    outStr = outStr.replace(/[źŹ]/g, 'z');
    return outStr;
}

Portuguese:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[áÁ]/g, 'a');
    outStr = outStr.replace(/[àÀ]/g, 'a');
    outStr = outStr.replace(/[ãÃ]/g, 'a');   
    outStr = outStr.replace(/[âÂ]/g, 'a');   
    outStr = outStr.replace(/[éÉ]/g, 'e');
    outStr = outStr.replace(/[êÊ]/g, 'e');
    outStr = outStr.replace(/[íÍ]/g, 'i');
    outStr = outStr.replace(/[óÓ]/g, 'o');
    outStr = outStr.replace(/[òÒ]/g, 'o');
    outStr = outStr.replace(/[õÕ]/g, 'o');
    outStr = outStr.replace(/[ôÔ]/g, 'o');
    outStr = outStr.replace(/[úÚ]/g, 'u');
    outStr = outStr.replace(/[üÜ]/g, 'u');
    outStr = outStr.replace(/[çÇ]/g, 'c');   
    return outStr;
}

Russian:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[аА]/g, 'a');
    outStr = outStr.replace(/[бБ]/g, 'b');
    outStr = outStr.replace(/[вВ]/g, 'v');
    outStr = outStr.replace(/[гГ]/g, 'g');
    outStr = outStr.replace(/[дД]/g, 'd');
    outStr = outStr.replace(/[еЕ]/g, 'e');
    outStr = outStr.replace(/[ёЁ]/g, 'jo');
    outStr = outStr.replace(/[жЖ]/g, 'zh');
    outStr = outStr.replace(/[зЗ]/g, 'z');
    outStr = outStr.replace(/[иИ]/g, 'i');
    outStr = outStr.replace(/[йЙ]/g, 'jj');
    outStr = outStr.replace(/[кК]/g, 'k');
    outStr = outStr.replace(/[лЛ]/g, 'l');
    outStr = outStr.replace(/[мМ]/g, 'm');
    outStr = outStr.replace(/[нН]/g, 'n');
    outStr = outStr.replace(/[оО]/g, 'o');
    outStr = outStr.replace(/[пП]/g, 'p');
    outStr = outStr.replace(/[рР]/g, 'r');
    outStr = outStr.replace(/[сС]/g, 's');
    outStr = outStr.replace(/[тТ]/g, 't');
    outStr = outStr.replace(/[уУ]/g, 'u');
    outStr = outStr.replace(/[фФ]/g, 'f');
    outStr = outStr.replace(/[уУ]/g, 'u');
    outStr = outStr.replace(/[хХ]/g, 'kh');
    outStr = outStr.replace(/[цЦ]/g, 'c');
    outStr = outStr.replace(/[чЧ]/g, 'ch');
    outStr = outStr.replace(/[шШ]/g, 'sh');
    outStr = outStr.replace(/[щЩ]/g, 'shh');
    outStr = outStr.replace(/[уУ]/g, 'u');
    outStr = outStr.replace(/[ыЫ]/g, 'y');
    outStr = outStr.replace(/[эЭ]/g, 'eh');
    outStr = outStr.replace(/[юЮ]/g, 'yu');
    outStr = outStr.replace(/[яЯ]/g, 'ya');
    return outStr;
}

Croatian:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[ćĆ]/g, 'c');
    outStr = outStr.replace(/[čČ]/g, 'c');
    outStr = outStr.replace(/[đĐ]/g, 'dj');
    outStr = outStr.replace(/[šŠ]/g, 's');
    outStr = outStr.replace(/[žŽ]/g, 'z');
    return outStr;
}

Romanian:

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[ăĂ]/g, 'a');
    outStr = outStr.replace(/[âÂ]/g, 'a');
    outStr = outStr.replace(/[îÎ]/g, 'i');
    outStr = outStr.replace(/[şŞ]/g, 's');
    outStr = outStr.replace(/[ţŢ]/g, 't');
    return outStr;
}

[plaszcz]

Polish version
     

Code:

function deLocalize( inStr ) {
        var outStr = inStr;
        outStr = outStr.replace(/[ąĄ]/g, 'a');
        outStr = outStr.replace(/[ćĆ]/g, 'c');
        outStr = outStr.replace(/[ęĘ]/g, 'e');
        outStr = outStr.replace(/[łŁ]/g, 'l');
        outStr = outStr.replace(/[ńŃ]/g, 'n');
        outStr = outStr.replace(/[óÓ]/g, 'o');
        outStr = outStr.replace(/[śŚ]/g, 's');
        outStr = outStr.replace(/[żŻ]/g, 'z');
        outStr = outStr.replace(/[źŹ]/g, 'z');
        return outStr;
    }

[reanimator]

i have now added accented vowels for the greek delocalize function:

function deLocalize( inStr ) {
var outStr = inStr;
outStr = outStr.replace(/[άΆαΑ]/g, 'a');
outStr = outStr.replace(/[βΒ]/g, 'b');
outStr = outStr.replace(/[γΓ]/g, 'g');
outStr = outStr.replace(/[δΔ]/g, 'd');
outStr = outStr.replace(/[έΈεΕ]/g, 'e');
outStr = outStr.replace(/[ζΖ]/g, 'z');
outStr = outStr.replace(/[ήΉηΗ]/g, 'i');
outStr = outStr.replace(/[θΘ]/g, 'th');
outStr = outStr.replace(/[ίΊιΙ]/g, 'i');
outStr = outStr.replace(/[κΚ]/g, 'k');
outStr = outStr.replace(/[λΛ]/g, 'l');
outStr = outStr.replace(/[μΜ]/g, 'm');
outStr = outStr.replace(/[νΝ]/g, 'n');
outStr = outStr.replace(/[ξΞ]/g, 'x');
outStr = outStr.replace(/[όΌοΟ]/g, 'o');
outStr = outStr.replace(/[πΠ]/g, 'p');
outStr = outStr.replace(/[ρΡ]/g, 'r');
outStr = outStr.replace(/[σΣ]/g, 's');
outStr = outStr.replace(/[τΤ]/g, 't');
outStr = outStr.replace(/[ύΎυΥ]/g, 'y');
outStr = outStr.replace(/[φΦ]/g, 'f');
outStr = outStr.replace(/[χΧ]/g, 'ch');
outStr = outStr.replace(/[ψΨ]/g, 'ps');
outStr = outStr.replace(/[ώΏωΩ]/g, 'o');

return outStr;
}

[codetwist]

Thanks, plaszcz and reanimator

[Dener]

Portuguese

Code:

function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[áÁ]/g, 'a');
    outStr = outStr.replace(/[àÀ]/g, 'a');
    outStr = outStr.replace(/[ãÃ]/g, 'a');    
    outStr = outStr.replace(/[âÂ]/g, 'a');    
    outStr = outStr.replace(/[éÉ]/g, 'e');
    outStr = outStr.replace(/[êÊ]/g, 'e');
    outStr = outStr.replace(/[íÍ]/g, 'i');
    outStr = outStr.replace(/[óÓ]/g, 'o');
    outStr = outStr.replace(/[òÒ]/g, 'o');
    outStr = outStr.replace(/[õÕ]/g, 'o');
    outStr = outStr.replace(/[ôÔ]/g, 'o');
    outStr = outStr.replace(/[úÚ]/g, 'u');
    outStr = outStr.replace(/[üÜ]/g, 'u');
    outStr = outStr.replace(/[çÇ]/g, 'c');    
return outStr;
}

[Patric Ahlqvist]

Ok, added following instead of old genSEF:

// generate SEF urls    
    function genSEF(from,to) {
        if (allowsef == true) {
            // MOD - 20070106 - Replaces characters specific to local language with available lowercase equivalent
            // Original code
            // var str = str.toLowerCase();
            var str = deLocalize(from.value);
            str = str.toLowerCase();
            // End of MOD - 20070106 - Replaces characters specific to local language with available lowercase equivalent
            //str = str.replace(/[^a-zA-Z 0-9]+/g,'');
            str = str.replace(/[^a-z 0-9]+/g,'');
            str = str.replace(/\s+/g, "-");        
            to.value = str;
        }
    }
   
function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[åÅ]/g, 'aa');
    outStr = outStr.replace(/[äÄ]/g, 'ae');
    outStr = outStr.replace(/[öÖ]/g, 'oe');
    return outStr;
}

Anything wrong so far ?? Thing is this do NOT att aa, ae or oe instead of the swedish carachters I'm typing, it's still a non value on every swedish carachter appearance... What am I doing wrong, I wonder ?

[codetwist]

@ Patric : Don't know :/ I even copied your script into my sNews and as well as those characters into new article  title - and sef was generated correctly. Only explanation I can imagine - that something is fishy regarding encoding/charsets You are using. Like, could it be that what is entered through web interface isn't the same that is put/get in JavaScript? Might even depend on local configuration? Have You tried to look at article editing source with something like Firefox Web Developer or similar tool? Does these characters looks correct and are they freely copy/paste-able unchanged? And does this work for agentsmith?

[codetwist]

Thanks, Dener

[Patric Ahlqvist]

Before ranting on... - Thanks, Codie for replying...

Something fishy alright... looking at the article form gives me this:

// generate SEF urls    
    function genSEF(from,to) {
        if (allowsef == true) {
            // MOD - 20070106 - Replaces characters specific to local language with available lowercase equivalent
            // Original code
            // var str = str.toLowerCase();
            var str = deLocalize(from.value);
            str = str.toLowerCase();
            // End of MOD - 20070106 - Replaces characters specific to local language with available lowercase equivalent
            //str = str.replace(/[^a-zA-Z 0-9]+/g,'');
            str = str.replace(/[^a-z 0-9]+/g,'');
            str = str.replace(/\s+/g, "-");      
            to.value = str;
        }
    }
   
function deLocalize( inStr ) {
    var outStr = inStr;
    outStr = outStr.replace(/[��]/g, 'aa');
    outStr = outStr.replace(/[��]/g, 'ae');
    outStr = outStr.replace(/[��]/g, 'oe');
    return outStr;
}

But, where, on gods green earth is this controlled ? I have UTF-8 everyfreaking where I can imagen.

I have downloaded index.php and snews.php and resaved them as UTF-8, I have charset UTF-8 I have sNews admin set to UTF-8 where the freaking dark hole of BeelzeBOB's rear end do I go wrong

*frustration make him go in a upright position, with stiff legs, banging forehead into brickwall*

[codetwist]

Ok. I'm no expert on these issues :/

Things to look for (just some guesses):
 - really correct meta tags (no contradictions, no excesses);
 - content language (not sure if this has anything to do with it but still);
 - sometimes web server might have some configuration that overrides anything in page (not sure about this either).

* and go easy on that brickwall, will You  :lol: *

[Fred K (agentsmith)]

This

var outStr = inStr;
    outStr = outStr.replace(/[��]/g, 'aa');
    outStr = outStr.replace(/[��]/g, 'ae');
    outStr = outStr.replace(/[��]/g, 'oe');
    return outStr;

makes me suspicious of your editor. Are you sure it edits and saves in UTF-8 mode? Normally, those wonky characters only appear when you're taking characters created in one charset and putting them in a doc that has a different charset without support for them characters.

As a second option (having exhausted all others) I'd chime in with codie about the server setup thingy.
All I know is that the mod works, under normal circumstances.

[reanimator]

do you have a copy of your php.ini ?

[Patric Ahlqvist]

Nope... but I've tried to change to charset on the tables using phpmyadmin from latin1 to utf8 but that did not work either...

Ok, my doing so far...

1. saved index - snews as utf-8
2. saved DB-tables in utf-8
3. UTF-8 charset in index

Nada worked. Saving the files in UTF-8 in editor (notepad++) added a "square" infront of charset in viewsource, and the whole shebang got sidewinded as in not shown properly. Saving the snews.php as UTF-8 in notepad++ made the SEF stop from functioningat all...

So now I'm back at square one. Both files as ANSI, everything else UTF-8, tables, index and so on...

God almighty... I need a beer so bad I would twist my nob if I thought it would come out beer form that.

[Joost]

I have downloaded index.php and snews.php and resaved them as UTF-8, I have charset UTF-8 I have sNews admin set to UTF-8 where the freaking dark hole of BeelzeBOB's rear end do I go wrong

Always be careful when saving php-files to utf-8 (without Bite Order Mark of course).  In very rare occasions, it can lead to unexpected errors. Very few documentation about this issue can be found.

I ran into this issue some weeks ago, while translating sNews. When I used the translation, I couldn't produce the '404 error, page cannot be found' page (new in 1.6). An empty screen was shown.
I have completely rewritten the translation (without copy pasting), tried saving it with Notepad++, with PSPad. The only solution is: Saving it as ansi.
So when not needed, save php files as ansi