(1.0.6 is already released, see task #1088)

I think everybody made his point very clear.

I support the proposal of converting po files on the fly before mo files as described by Tobias. Continuing to argue about that would probably cost way more time than maintaining such conversion will ever do.

So we'll implement that; and translators that do not use utf8 will not notice anyway.

MR: "... what cost us nothing ..." -- Not so sure. It will certainly cost the compatibility with non-UTF-8 people (including myself) or development effort to maintain this compatibility.

Look, there is no such thing as a pure gain. There is always trade offs: speed for storage, time for money, something for something else.

• UTF-8 version: presumably more quick, takes more storage (no pure gain...), requires either UTF-8 po-files encoding or on-the-fly conversion, requires UTF-8 <-> 8-bits conversion on 8-bits systems (from developer/translator point of view)

• 8-bits version: less storage, but slower (worth noting, though, that gettext uses the hash tables internally, so the lookups and the conversions are done only once), compatible with 8-bits system (from developer/translator point of view).

MR: "... the failure would mean having what you propose to have anyway ..." -- If only this were true... The failure would mean having what I propose minus the time, that could have been spent to improve (and/or maintain) something else.

P.S. Mathieu, could you give me a sign, when 1.0.6 is ready? I would like to try to install it on a clean machine (and to try some unusual configuration).

Mathieu Roy: "I'm not sure that I understood the "against" point of view." -- It is not really "against". It's just the question: Why should we impose something onto the translator, when the system does not relly need that?

In other words, I do not think that this difference will ever make it to the differences that make the difference; I do not think this is worth the effort, and it looks just silly.

Besides, if someone wants this difference in performance (after have found the computer on which this difference will be showing -- which is difficult task by and in itself!), that he must be able to do it locally (and eevntually report the success story).

If the system in general does not require this kind of optimisation, we should not impose it (but this kind of optimisation should not be forbidden to the administrator, if his system requires it).

Tobias Toedter: I wouldn't do this. It is just one more point of failure, and one more piece to maintain.

"is perl really needed? Cannot iconv simply do it?"

I would use iconv for the actual conversion, yes. But I need to change the msgstr header for the .po file as well, because the encoding is specified in there. That would still remain KOI8-R, for example, even though the ru.po file would already be in UTF-8. Obviously, this leads to terrible results or even failures.

Thus the little header snippet has to be replaced as well, that's where perl comes in handy.

How about this:

Leave the .po files in the encoding the translator prefers and convert them on the fly into UTF-8, before the .mo files are generated. This could be done in the target "make %.mo". Afterwards, those changes are discarded and the .po files stay cleanly in their original encoding.

This would require a little bit of black magic in perl, but it shouldn't be too hard. :-) I'm willing to set this up for release 1.0.7.

I guess this suits both parties, right?

You are right about the mo-files encoding, Tobias.

The question is: how much CPU cycles will be saved by encoding the .po-files in UTF-8? Is it worth it? It is hard to tell without profiling, but I /guess/ the difference is negligeable.

Tobias Toedter <toddy>: "it's certainly possible to require the .po files to use UTF-8".
-- If I remember correctly, po-files do not have to be in UTF-8; any 8-bit encoding is converted automatically, when you "compile" po-file into mo-file (that is why there is the line "Content-Type: text/plain; charset=KOI8-R\n").

"I don't know what you mean with this. If you refer to the changes I made WRT ngettext, I can assure you that there are no UTF-8 issues mixed in the current code. Are there any other problem areas? "

Ok. My mistake.

"But I'm afraid that there might be some small fraction of database content which is not pure iso-8859-1. This content will most probably not display correctly after the conversion."

Will have to come up with an update script that check content and convert appriately. We cannot propose an upgrade that could cause serious data loss.

Apart from that: there's no problem in publishing a new release just for gettext stuff.
But since it's serious change, this must be done on branch in any case, not on the trunk.

So let's say by the end of the week we publish the 1.0.6 if no serious problem shows up. And after the release, you create a branch and open a related task on the task manager, and the 1.0.7 will only depends on this task (or eventually important bug fixes).

DIG proposal is interesting. Unfortunately, since the whole utf8 stuff already happened on the trunk, it do not know what we should do exactly.

"3. Convert the file from ISO-8859-1 to UTF-8, using icon"

- Are we sure have only iso-8859-1 content?
- Are we sure only utf-8 will be put in the database afterwards?

"Any necessary conversion is done on the fly."

But Savane will not convert stuff to utf8 each time it loads a page, will it? If so, isn't it resources (cpu) waste -- since we'll want utf8 anyway, converting everything once and for all would save resources?

"Would it be possible to let the release task for 1.0.6 depend on this bug as well?"

If this item can be closed by the end of the week, that should not delay the release. Would it be possible?

I agree with you, Silvain. I think that enabling of UTF-8 should be possible without too much trouble. I further think that we should do this before we release 1.0.6, as it would be a great opportunity to get a smooth transition.

WRT the database, it is indeed not that trivial as I first thought. Unfortunately, the CONVERT() function is only available in MySQL 4.1 or greater and is therefore clearly ruled out. To do the conversion nonetheless, we have to do it "manually".

But here's the good news: I thought a lot about the way how to do it, and I think I discovered a really easy way which should have no side-effects:

1. Get a dump of the entire savane database
2. Store this dump in a temporary file
3. Convert the file from ISO-8859-1 to UTF-8, using iconv
4. Dump the converted file back into the MySQL database server

Some remarks about step 3: All the "metainformation" for MySQL, like table structures and INSERT commands etc. are plain ASCII, so those are not affected by the conversion. I cannot think of any error which could leave the database in an unusable state, but since this is obviously the most critical task, comments are very welcome.

OK, on we go: The site-specific files can easily be converted using iconv. No problem here.

About gettext/ngettext: The .po files can be in whatever encoding, it doesn't matter anymore. Any necessary conversion is done on the fly. Moreover, the adaptions needed for savane to put out real UTF-8 are trivial and really minor. I've tested them already, those fixes work smoothly.

I would like to point out that I'd really appreciate this bug to be solved before 1.0.6, and I don't think that the changes needed are too intrusive to be performed at this (relatively) late point in the process.

Mathieu, what do you think? Would it be possible to let the release task for 1.0.6 depend on this bug as well?

I would just like to pinpoint that Savane is a separate activity from Gna!, hence not everybody has access to the Gna! test install.

Aside from that, regenerating the .po/.mo files should not be necessary, as gettext calls iconv at run time to translate the messages in the appropriate charset on the fly.

Looks like we all have to carefully study CONVERT() now ;)

I confirm that GNA! test install now works well in Russian (before there were some problems with double encodings).

As a side note: I would prefer to keep .po-file (at least, ru.po) in 8-bit encoding (KOI8-R), and to convert it into UTF-8 encoding only when .mo-file is being generated.

"I never said there could be a misconfiguration at Gna!, sorry if I wasn't clear"

I never thought you said that. I was just saying that since the problem occurs on both installs, it is unlikely do to an uncomplete chroot install.

"The encoding used during form submission must be the encoding of the page, unless I'm really missing something. If your page is UTF-8, you get submitted text in UTF-8."

That's my experience too.

"header('Content-type: text/html; charset=utf-8')"

Ok, that's necessary in case a configuration cannot be site-wide. For http://www.gna.org, it is not really necessary but there's no reason not to add that in Savane.

"site-specific/*.txt files "

Cannot we just convert these to UTF8?

Ok, so apparently:

- one can redefine the encoding used by adding ".UTF-8" at the end of the locale name. The gettext bind_textdomain_codeset appeared in PHP 4.2 which is above debian stable, so I didn't test it. When you do that, you also have to set your owm content-type header via header('Content-type: text/html; charset=utf-8').

- include() doesn't seem to take care about encoding. I tried to get force PHP to use UTF-8 via iconv_set_encoding, but for reasons I could not figure out yet, this is messing up https. This means that if we switch to UTF-8, old site-specific/*.txt files will not be valid. Since the page was supposed to be under UTF-8, from the start I don't think this is a big issue.
We could consider trying to guess the files' encoding and do the appropriate conversation, as do Emacs and Jikes, but that would be resources consuming and kinda complicated, I guess.

To sum-up: 1) $locale.='UTF-8'; 2) header(...).

As for texts included from the database, somebody has to check:
- the encoding used during submission by html form
- the encoding used when adding to the database (it may be PHP's internal encoding)

Unassigning the item because I'd be glad if someone else was taking it over.

"At Gna, setting the http header (via apache) default charset to utf-8 ended up in broken output (weird characters)."

This (latin1 interpreted as utf-8) is the opposite of my german
problem (utf-8 interpreted as latin1).

"A thing that could matter is the locales supported by the differents systems. http://www.gna.org support all locales shipped by debian
(both utf8 and iso for both de and fr locales)."

That what I did. At Savannah, all locales were generated. Provided we copied them right in the apache root..

I'll try to make tests in a more normal install tomorrow.

"Doesn't it respect the original files charset (po files and php
files?). If so, how should we recode these files to make sure they are all utf8? I guess there should be documentation about it (I'm completely clueless about that)."

gettext: Maybe we could use the bind_textdomain_codeset PHP
function. I could not find "good practice" for PHP+charsets, that's a pity :/

files: source files should have stuck to us-ascii; your french
templates (.txt) at Savannah, though, are latin-1. 'iconv' or 'recode' is ok indeed, unless PHP is automatically recoding files during includes.

There is also the problem of database data: project description,
tracker comments... I don't know how that is managed.

Mathieu: just try `iconv --help' in your command line. `iconv' should be enough for converting between charsets.

This has forced me to convert my download page (http://download.gna.org/quarry/) to ISO-8859-1 (was in UTF-8.) Before it was HTML 4.01 Transitional, and the charset was picked up correctly by the browsers, but now it is XHTML 1.0 Strict and it seems that the browsers now prefer to use the charset from the HTTP headers.

It seems that the server at home.gna.org sends UTF-8 in HTTP headers, so my UTF-8 XHTML pages there are displayed just fine.

So, I guess gna.org server should sort of adopt configuration of home.gna.org one.

Since I have some issues at Savannah (https://mail.gna.org/public/spam/2005-01/msg00428.html - hmm, what's this doing in the spam???), here are some information:

- PHP is not sending any charset information by default; it can be configured to set a site-wide charset that will be added in the Content-type HTTP header.

- If no charset information is set by PHP, Apache adds Latin1 in the Content-type header by default (the charset info is part of the Content-type header); that can be changed too. It should not be removed for twisted security reasons, unless the document is served as application/xml. Check the link referenced in the Apache default configuration file (search for 'charset').

- Savane's template include an XML declaration that specify that the charset used is UTF-8; this declaration has a lower precedence than the HTTP header. Also, this declaration is normally not taken into account if the XML (here XHTML) document is served as HTML (text/html - which is the default). It is taken into account if the document is served as application/xml.

At this point, solutions include making PHP (or Savane's template) set the content type: either it sets the content type as application/xml (and the encoding becomes what is specified in the <?xml?> processing instruction); either it sets the charset itself to, say, UTF-8.

Now, it is not that simple, because we do not know what encoding PHP will use, independently of what the declaration we used.

Especially, my test install and Savannah have a strange behavior: when using the French locale, output charset is latin1, and when using the German locale, output charset is utf-8. However, at Gna!, output charset is latin1 for both locales. All those are running Debian stable.

Note that localized texts are supposed to be automatically converted by gettext to the charset specified by the user in his LANG/LC_ALL/... environment variable (at least it works for console applications).

So the first step is to understand what charset is actually used by PHP (or by gettext?) for its output and how to set it. The second step is to choose how to describe it.

In fact, it's not so simple. Savane pretend the content is utf8 while it is not.
Normally, apache override this erroneous information. But if apache is changed in this regard, we end up with broken output.

1. Deciding whether a bug should be closed or not is internal management of the project. Some bugs trackers allow submitter to close item. But Savane have a solid user base against such approach and no real user base in strong favor of it.
(Savane have a specific role model, being able to close item is a key point of it).

2. I see no exact rendering issue. It is true that in fact the utf8 charset will be ignored, due to the http daemon. That's more a Gna issue than a Savane one. Speaking for gna, it is unclear what we should do about it considering we run debian stable.

Sorry, I didn't check it and relied partly on my previous memory. Apparently, Gna! pages are now in UTF-8, so converting entities into characters is not problem.

This bug should be closed as invalid.

BTW, why am not able to close it? I'm the one who opened it, other bug-tracking systems allow bug originator to close it.

In my project's description I have an em-dash as HTML entity. When I edit this description, the dash is displayed as a character in the edit field, not as an entity. This is because Savane does not convert "& mdash;" sequence to "& amp; mdash;" when generating editing page source (extra spaces to prevent potential problems ony.)

I remember there was a PHP function for this, but don't remember the name of it.

There may be other places where this problem shows up.