For me it works. Feel free to test the branch and report bugs, mentioning the branch as affected release.

A preview would probably be a good idea (but I think we should keep that for a later release).

The ability to edit an already posted message raised lot of issues and I personally think we should avoid it. It does not ease discussion when there is no way to be 100% sure of who said what. :)

(test please ignore)

I'd like to start testing this great formatting stuff on gna, but bug #5346, bug #5347, bug #5348 and task #2878 are blocker for me on that regard. I could start try to fix them but you probably have a clearer view on how to do these things.

> It is maybe due to the fact that you called
> utils_basic_markup() (basic), but now all the line breaks are
> gone.

Well, the basic markup doesn't care about paragraph formatting, so it doesn't add any line breaks or paragraph breaks.

Where did you discover this problem? I thought that most uses of utils_make_links() should not add line breaks.

So, the function utils_make_links() is gone now and has been integrated into utils_basic_markup().

I've replaced every call to utils_make_links with the appropriate call to one of the markup functions. This should also resolve bug #5300 and bug #5301.

It is very funny, because "basic" was the first thing that came to my mind but the idea of "rich" made me change it :)

So let's go for full/rich/basic (I admit that "poor" may seem pejorative)

Note that I do not think that "poor" should allow verbatim and maybe even not nomarkup.

We want to allows users to specify verbatim where they are supposedly to rightfully post code and things like that -- definitely not in CC comments.

And nomarkup should be for us, for site admins that upgrade an installation and wants to avoid some issues with old items that would cause trouble with the formatting mechanism.

I've just reverted the hack to enable utils_wiki_markup() from utils_make_links().

The effect is now that utils_wiki_markup() gets called for the details of an item, but nowhere else. In the long term, all calls to utils_make_links will be replaced with utils_wiki_markup (well, at least where appropriate).

> without needing to change ~200 lines which call utils_make_links. As you quite
> rightly pointed out, not every call to utils_make_links can unconditionally be
> replaced by utils_wiki_markup, that's where this bug comes from.

Would it be easier to have only one function used everywhere?

I think it would be acceptable to have bold in a comment of an attached file.

The issue is about restricting formatting.

On original news items and project description, we want everything (headers and hr)

On items comments and original submission, we want everything but headers (I think hr is fine for comments)

On shorts comments like CC comments or attached files comments, we only want the most basic things.

I do not thing any <p></p> should be added on a simple string. Why would it be so? Is it necessary?

> (utils_make_links is used in many places for simple strings,
> that cannot be surrounded by <p>)

Alright, that's a bug. However, please note that the current formatting is kind of hackish, it just does the markup from within utils_make_links, that's why the <p> tags are added.

In the release version, we would call utils_wiki_markup where appropriate (i.e. replace the call to utils_make_links), and that function in turn should call utils_make_links, not the other way round. This will enable us to have the links converted where needed, without additional formatting applied, by calling utils_make_links directly.

I've applied this hack so that we're able to see the effect of markup working, without needing to change ~200 lines which call utils_make_links. As you quite rightly pointed out, not every call to utils_make_links can unconditionally be replaced by utils_wiki_markup, that's where this bug comes from.

Hum, I would be over-cool if quoted content like

> I quote the previous message
> bla bla

was in a different color (lighter), as it is often in nowadays email clients.

That should not cost much, and we do not need overcomplicated things (for instance color change different if it is > or >> or >>> etc), just a basic color change.

About the numbered lists: you're right, we should switch that. I'd like to keep '*' for unnumbered lists, though. How about using '0' as the start of a numbered list, like this:

1. list item
   1. nested list
2. next item

> I know Tobias that the wiki syntax in proposed in my original
> message was not doing it that way, but I guess it's time for
> fine tuning isn't it :)

Sure, that's what testing is about ... :)

> = adad = doesn't work on my test install

That's correct, we support headers from level 3-6, i.e. with three to six equal signs on each side.

> verbatim seems still incomplete regarding fonts but it
> properly disactivate the markup.

> About the tag styles, sorry to be picky :P, while I like
> #verbatim#, I dislike #/verbatim# (seems weird).
>
> How about
>

Hm, I think I like that best. It's now implemented, and I've changes its name to and .

Mathieu, the wiki markup hasn't been activated yet. I just committed a fix for this, although the fix needs to undergo some more thoughts. Currently it's a quick and dirty hack. It must be solved properly before the release.

However, the wiki markup should work now ... could you test your installation of the branch again?

About tags:
----------------

About the tag styles, sorry to be picky :P, while I like #verbatim#, I dislike #/verbatim# (seems weird).

How about

...

or

#+verbatim#

#-verbatim#

#+nomarkup#

#-nomarkup#

About markup behavior:
-------------------------------

The reason I'd like verbatim to print things as they are is the idea that people that will use this will want it to really do it's best to print things as they were "verbatim". And fixed-fonts and all just make this possible.

On the other hand, it may be useful as well to have the nomarkup, just in case the only thing that matter is to get unaltered characters but while the output style does not matter.

Finally, I don't want to go for <code> and all that tags widely used so far, first because they refer to things known only... by people that already knows (popularity does not mean being popular to an elite), second, the point is to avoid to alter content by mistakes, and to avoid that the best is to avoid XML/HTML like tags that are expected to use by people not especially aware that using them alter their content.

As I already said I think, I would not like having someone copying/paste a bit of HTML "aacc <code>ada</code>adad" to have it's content altered just because he did not knew that Savane would convert "<code>".
With alternative tags that are not referring to a know markup language like HTML, I guess we drastically reduces the odds that it happens.

I still think there should be something like <code> HTML tag. Tags like <font>, <b> etc. are deprecated in HTML because they are purely presentational and carry no semantic meaning, unlike <code>.

You could also have <pre> analogue like in MediaWiki: if a line starts with a space, whitespace is honored in it. This is quite convenient, but on the downside, can be pretty surprising to an unaware user.

Does #verbatim# tag imply fixed-width font? If it does, then it makes escaping unintended markup clumsy, else there is a need for code-introduction tag.

In either case, I propose #code# or some similar tag. It should turn on fixed-width font and be translated to <code> HTML tag, which carries semantical meaning. Whether or not it should imply #verbatim#, I'm not sure. Implying is convenient, but orthogonal tags give greater flexibility...

> I cannot attach a 50 MB file to this task.

Mathieu, I was only kidding, of course. It doesn't make sense to store a complete database dump in the very same database ...

I also don't need administrative data like users or passwords. Could you just send a dump of e.g. the bugs table to me -- or better, upload it somewhere so I can download it?

With regard to the headings, you're right. We should support them from level 3 on.

I cannot attach a 50 MB file to this task.

I cannot dump everything as I get it would be unwise to put somewhere outside of the backups servers dump including passwords (ok they are md5ized but...).

I'll make a dump of only items, if that's fine.

As said, I have real issues with quotes. These are the cause of probably half of the bugs I encountered during savane development, mostly due to escaping things. I'd really love to avoid these nasty things.
Also, somehow it hurts me to get quotes for a usage that is note quotation :(
Finally, I'm definitely sure this will lead to major headaches regarding possible false positive, as in the code, quotes are quite common. '' is so alike " that is is really an open door to problems.

No?

While wikipedia is definitely a very nice piece of collaborative work, their choice regarding technical things may not be so wise to be followed. I gave a tried to mediawiki for my local webserver at home. I use a wiki to work. It was damn slow for nothing, nothing by comparison to Kwiki, I had to drop it only a few minutes after installing it. It's never a good sign when a piece of software is litterally unusable while you havent started using it at all, when it is not really doing anything.

The big difference is the fact that ''' does not really means anything. You could use ''' to mean whatever you want, it has no real background.

adad and adada are way more common, have an history from the days when no mail client was handling html.

And frankly, it is easy to remember ada = emphasis and aada = bold, while remembering that ''' is bold and not emphasis, and not the contrary, is definitely getting complicated.

Finally, quotes are already used in so many ways and so many places, I guess it is more safe to avoid messing with it. And if we get back to litteracy, quotes definitely means something different than bold for instance.

Doubling signs indeed seems a good way to avoid false positives. But I'm a bit annoyed in fact with the text version output of it that will be in mails. Maybe it will be a bit ugly. Maybe not.
So maybe we could give the non-double version a shot. I don't know.

> I still don't like the idea very much to add some special tags to every DB

> entry. I think it would be a cleaner solution to disable the wiki markup for
> those entries completely, even for future entries. We could then allow the
> markup only for project descriptions and recipes.

No no, in many cases, in comments and elsewhere, it would be truly nice to be able to do some formatting stuff. Indeed, the wiki have to handle 2 levels of formatting: for recipes content, news contents, project description ; for comments and usual items details. In the first level, headers and hr should be allowed, not in the second level.

If we don't go on special tags for every db entry (I admit you are right, it is not an uber-clean way to do things), at least we should double markup signs, *strong*, //em//, _underline_.

And in this case, yes I guess we should provide an update script that would use the wiki formatting regexp to detect whether it would do any conversions in previous items. And if it does, it should ask whether a #verbatim# tag should be added around the incriminated words (not the whole item details or comment) so the admin when upgrading his installation would be 100% sure that after the upgrade there are no uncatched issue, no bits of code altered.
(asking him to do everything by hand in phpmyadmin is a no go even if only 1% of the items are affected. On LCG Savannah, it would mean editing more than 130 items!

On another topic, I've just realized I'm a bit annoyed by the underline idea and the /complexity/. Normally, in a word where you study litteracy, you are supposed to use underline to mean italic when you are writing by hand. Underline is supposed to be only a clone of italic more doable for a human. You frequently use this when you are using foreing langagues words in your text.
So maybe we should not contradict this principle (that definitely ring a bell to me) and use _adada_ for the italic and skip the underline idea.
This will be more logical for a typographical point of view. And it saves us the frequent case of http://mywonderfulurlwitha/andanother/
What do you think?

> Frankly, I do not want to simply do test for this. We wont be
> able to check the 13000 items posted at LCG Savannah and we
> cannot publish a version that may alter inappropriately old
> item. So even if we do test and dont detect problems, we'll
> still have to add tags for the old item so they are not
> affected by the wiki formatting because they were not meant to
> be formatted this way.

Would it be an option to write a script that checks this and reports possible problems? The admin could then fix those before upgrading. I imagine that those errors will be only very few. That's why I'd like to run some tests on the Savane DB. If it turns out that there are many problem cases, we must not perform that markup.

I still don't like the idea very much to add some special tags to every DB entry. I think it would be a cleaner solution to disable the wiki markup for those entries completely, even for future entries. We could then allow the markup only for project descriptions and recipes.

> Why not. But in this case, shouldn't we use double signs
> everywhere for cohesion sake?

Hm, we could do that. However, I like this syntax better than this *syntax*. We could also allow both, since the single markup characters already work ...

I don't like the idea to display each and every entry ever submitted in verbatim, especially as we're not doing this currently.

The text I'm submitting right now for this task won't be displayed verbatim, but filtered through nl2br() instead, so I get my linebreaks like they were intended. Also, if I use a reference link to some bug number, it'll be converted as well.

The wiki markup function currently does correct markup of paragraphs (i.e. not just <br />, but <p></p>), which will display nicer in the browser. It's also more to the intention of the submitter.

Having all texts displayed with a fixed-width font makes reading quite ugly, in my opinion.

I suggest to try this function with real data from the trackers and see how the data will be marked up, so we can discuss how to solve possible problems.

With regard to the hyperlinks, I like the proposed syntax as well. However, I suggest to use double brackets [[link]] instead of single brackets, so it's easier to match with a regexp.

Well, the way I see it, we need a tag that means verbatim, which means that the content of a message wont be altered. Links will be added whenever useful (like "tracker #nnnn", or "http://url") but that's all.

And all previous items that are not cookbook items or "project description" (as shown on projects main page) should be altered so their content is considered as verbatim (we just have to put the verbatim things at the end and begin of each comment and details).

We don't have to provide a formatting that is backward compatible with text-only content, we just need to convert HTML whenever it was added as HTML. Everything else must be added verbatim because in the mind of people that submitted it, it was meant to be shown verbatim (and that's how it worked until now).

Do you get my point?

For the verbatim tag, I think we surely need something verbose and unusual, like
#verbatim#
bla bla
bla bla
#verbatim#

And for this, we should also update the css so the content is printed with fixed-font and ideally that it is true verbatim - indentation would be respected.

There's only one issue with that: true verbatim would not allow us to do html for instance for replace "tracker #nnn" with a hyperlink. There's only two options in this regard:
- doing only partial verbatim, keeping things in html allowing us hyperlinks. Backward compatible without effort but break indentation and makes our verbatim mode awkward
- doing true verbatim but having the update script that will add the #verbatim# (whatever the string is, it is just an example) at the begin and the end of items content also adding the same string before and after each links -- I think this is likely to cause plenty of false positives
- doing true verbatim but having also a partialverbatim tag, that would not be advertised but be the one that would be used by the update script, that would simply add #partialverbatim# at the end and begin of each old item content. This would tell to the wiki format function to be behave how savane behave in until now, by altering content to put hyperlinks but not applying wiki format.

The late option is the only sane option, IMHO.

Side note: details and comments content should be in "text-align: justify" as soon as it's not inside #verbatim# or #partial verbatim#.

One more point. Maybe the different emphasize formatting should use doubled characters (e.g. //italic// instead of /italic/) to avoid most false matches?

Maybe we could use simplified MediaWiki syntax with complicated features stripped off? I personally hate camel case and would appreciate nice links with variable text part like MediaWiki provides. Camel case links are especially awful for languages with reach morphology, like Russian or, perhaps, German, because they make sentences grammatically incorrect even if you ignore the run-together words.

Simple comment: I think we should put the wiki format in the database, and do the conversion on the fly only.
It will be easier to extract the data in text (for instance when sending text only notifications.
Indeed, it means that we will have to run utils_wiki_format() on each relevant output, so I guess this function has to use regexp in a very cheap way (using preg_*() etc).
No?