tips : website in UTF-8 :: Zikula Community :: Support at your fingertips

Try Zikula 1.3.1!New core, modules & more

Download Zikula!Version 1.2.8 (stable)

tips : website in UTF-8

Print topic

mumuri

Rank: Developer

Registered: May 17, 2004

Last visit: Apr 08, 2010

Posts: 386
Posted: Jan 13, 2009 - 11:45 PM

So first, UTF-8 is an encoding which allows you to display complex characters (like chinese kanji or special character like in the bottom of charset tools in windows).

If you wan't to convert your website to UTF-8

If I consider that you already have converted your database in UTF-8

1- edit "language//core.php" and change the charset constant

Code
define('_CHARSET','UTF-8');

2- edit "includes/pnAPI.php"

after

Code
if ($stages & PN_CORE_TABLES) {

~line 322

add

Code
$query = sprintf('SET NAMES \'%s\'', 'UTF8');
$result = DBUtil::executeSQL($query, -1, -1, true, true);

3- edit config.php and change the dbcharset

Code
$PNConfig['DBInfo']['default']['dbcharset'] = 'utf8';

That's all

Notes:

there was once an article , which explain how to do this
http://community.postnuke.com/index.php?name=News&file=article&sid=2831

i don't explain how to convert the database, because it's not zikula specific and that there is already a lot of tutorial on the net to do that

But it doesn't seem to be online anymore.

edited by: mumuri, Jan 14, 2009 - 05:41 AM
jguille2

Rank: Registered User

Registered: Jun 25, 2008

Last visit: Jan 14, 2009

Posts: 12
Posted: Jan 14, 2009 - 01:20 AM

Hi,
Very clear and useful... I'll try to add something (maybe questions).
1-On /config/config.php, line 55, i'd change 'latin1' for 'utf8'. I thought that it was important. If you haven't done, maybe not. Perhaps it's only important when you run the install.php and don't have a database created?
2-When I've made translations (to Catalan, with our Ã , Ã©, Ã¨,...) I found a problem. Zikula files are in utf8. Then the text of the 'defines' (in .../pnlang/eng/...) are utf8 too. In English there isn't Ã , Ã©,... and, then, all works fine. When I translate files (.../pnlang/cat/... or .../pnlang/spa/...) I must change utf8 to latin1 in the PHP file to work ok.
Then, when somebody make the change to utf8, if he has diferent languages, he must change all the .../pnlang/xxx/... to utf8 from latin1.

3-A question. In the /config/config.php, where defines 'latin1', says 'will be changed to 'utf' after PN 0.8!'. I hoped Zikula would be in utf, but it isn't. When this change?? I thought it was more complicate, but if this first post it's right, the change it's very easy.

That's all. Thanks,
Joan
mumuri

Rank: Developer

Registered: May 17, 2004

Last visit: Apr 08, 2010

Posts: 386
Posted: Jan 14, 2009 - 01:44 AM

for config.php , you're right, i forgot to mention it.

In my case all files are in UTF-8 (language and PHP file), and don't have any problem, that strange.
jguille2

Rank: Registered User

Registered: Jun 25, 2008

Last visit: Jan 14, 2009

Posts: 12
Posted: Jan 14, 2009 - 04:05 AM

Hi mumuri, what's your language?
Latin1 and utf8 are very similar in the ordinary characters and so, there is no problems in english. If you use Ã¡,Ã ,Ã©,Ã¨,...,Ã±,Ã§,.. then you have problems. In our 'catalan modules' (intraweb project), in pnlang files, we use latin1 for spa and cat (spanish and catalan) and utf8 or latin1 for eng (usually latin1 because we translate from catalan files).
nestormateo

Rank: Team Member

Registered: Sep 06, 2006

Last visit: May 09, 2010

Posts: 2411
Posted: Jan 14, 2009 - 04:28 AM

I've heard from Chris that it's required to have the DB in UTF-8, the config and the templates. No more is needed to have the site in UTF-8. I guess there's a Wiki page or something (a Wiki page or an article) in the german community, can a german link that stuff please?

--
- Mateo T. -
Mis principios... son mis fines
jguille2

Rank: Registered User

Registered: Jun 25, 2008

Last visit: Jan 14, 2009

Posts: 12
Posted: Jan 14, 2009 - 05:04 AM

Hola Mateo (Cuanto tiempo! Supongo que la tesis y demÃ¡s ha ido bien... Ya hablaremos)
Ok, I think is important to document this point. It seems easy but we need to take care...
For example, these post in this forum:
-I see the meta tags of this page: it seems to encode in iso-8859-1
-But my browser set this page in UTF-8 (automaticaly). Why? Is it wrong the tag definition? It's clear there are something (templates,...) in UTF-8 that makes browser to see this in UTF-8.
-But the content of the posts (bd) are in latin-1. This makes than in my posts, i write a (with accent) but you see Ã¡. If you change manually in your browser the iso8859 codification, you will see this characters Ã¡ Ã Ã© Ã¨...
---
Joan
slam

Rank: Team Member

Registered: May 03, 2004

Last visit: May 31, 2010

Posts: 509
Posted: Jan 14, 2009 - 05:41 PM

Your database can have any given encoding, also the database connection - both actually do not matter. However, your database tables and the content needs to be encoded in UTF-8, as do all language files and all templates. There are several tools around to do that, depending on the operating system used and your console skills. You do not need to convert file names, just the content of those files. You should convert file names of uploaded files, though.

Be careful with mixed encoded content (which you might find in news, comments, forums, PM, etc.), you need to do the conversion there manually. That can easily become quite a lot of work, so for bigger sites seriously consider to create a public read only archive of the site, and start a new one in UTF-8.

The file language/$yourlanguage/core.php needs the change already described above, and you need to do that for all languages! Also the file /config/config.php needs editing as described above. But: No other core files need to be hacked in Zikula 1.1.x, specially not includes/pnAPI.php (if that would be needed a bug report should instead be filed in Cozi).

Empty pnTemp/Xanthia_cache and pnTemp/pnRender_cache.

Your milage may vary with some more complex 3rd party modules/blocks/themes, some optionally already might provide UTF-8 language files, while others need manual fixing. You need to take special care for RSS feeds and all content which might be shown where UTF-8 is not available.

Starting with 1.2.0 Zikula will default to UTF-8.

Last word: Browsers need to be able to auto-detect and show UTF-8, and your operating system needs UTF-8 fonts ready for that. Both is true for most modern browsers and operating systems, though, but do not forget this when developing for big organisations.

Greetings,
Chris

--
an operating system must operate
development is life
my repo
nestormateo

Rank: Team Member

Registered: Sep 06, 2006

Last visit: May 09, 2010

Posts: 2411
Posted: Jan 15, 2009 - 08:09 AM

Hi Joan! (yeah, thesis finished successfully and now working in zikula.es )
The header problem was finally fixed. Server-side settings affects the encoding too.

1.2 will be UTF-8 by default and if i want it i must convert all the content?
well, i guess that in many cases i will preserve the ISO-8859-1 then

BTW Chris, recommended tools to do this job?
i've tried to play with my CLI and iconv with no success
any good suggestion to convert content with "ease"
which is your way?

--
- Mateo T. -
Mis principios... son mis fines
hilope

Rank: Developer

Registered: Aug 05, 2003

Last visit: May 06, 2010

Posts: 187
Posted: Jan 15, 2009 - 09:30 AM

since 2005 i'm working with UTF-8 for postnuke/zikula and i'm happy that version 1.2.x will be the first version with UTF-8 support.

for additional information you can read my articles on german community:
google translated links (so read creative )
1. http://translate.google.de/translate?prev=&hl=de&u=http%3A%2F%2Fsupport.zikula.de%2Fmodule-News-display-sid-743.htm&sl=de&tl=en
2. http://translate.google.de/translate?prev=&hl=de&u=http%3A%2F%2Fsupport.zikula.de%2Fmodule-News-display-sid-1036.htm&sl=de&tl=en

an important point is the server setting in .htaccess:
AddDefaultCharset UTF-8
AddCharset UTF-8 .php
AddCharset UTF-8 .htm (for "htm" short URL setting)

perhaps this will help

edited by: hilope, Jan 15, 2009 - 10:32 PM

--
| scribite! | TNGz | piwik | zweet |