Add last part of task #4126: the script that will use sa-learn so spamd use bayesian filters

(Browse SVN revision 6425)

The currently done part can already be tested. The remaining learn script for spamassassin bayesian filtering is not essential.

I wont test it at Gna! because so far we have no spam problems here yet and so there is no reason to delay posts from anonymous and increase spamd load. If you run an installation affected by spam, you should try.

Add SendTrackersDelayedNotification() to deal with delayed notifs (task #4126): able to send notifs exactly how the frontend would have sent it (FIXME: MailSend() is unable to force the msg-id)

(Browse SVN revision 6406)

Add second part of the backend regarding task #4126: monitor the queue, remove stuff in there for too long (> 15 minutes)

(Browse SVN revision 6400)

Add first part of the backend regarding task #4126: now handle items put in the queue by the frontend and pass them to spamc. Map spamassassin score to something sensible in regard of savane own scores:
# Found out a reasonable score for the item, considering that the current
# score equal to default + 5.
# if the score is below 0, score = 0 (-8)
# if the score is between 0 and 2, unchanged (-5)
# if the score is between 2 and 3, increment of one (-4)
# if the score is between 3 and 5, increment of two (-3)
# if the score is between 5 and 7, increment of three (-2)
# if the score is between 7 and 9, increment of four (-1)
# superior to 9 , increment of five (untouched)

(Browse SVN revision 6398)

Do the frontend part of task #4126: put newly posted items in the spamcheck queue, according to conf (all or anonymous post - project members are always excluded from this process)

(Browse SVN revision 6397)

Add new spamcheck queue table (task #4126)
Wondering about:
-- should we put the summary in this database or extract it when checking?
-- * duplicating data means we do bigger inserts
-- * extracting when checking means doing more SQL requests

(Browse SVN revision 6362)

So I thought yesterday about this. The way I see it:

• Site admin configure what kind of content should be checked: anonymous post or anonymous + logged in post
• On the frontend Each time a content to be checked is posted:
  • it get an additional temporary) +5 in spamscore. So no notifications are sent and the comment is virtually invisible. Project admins can however already unflag it as spam, if it is legitimate content.
  • it is added in a new table trackers_spamcheckqueue that record the timestamp of the add
• On the backend, there is
  • a script (sv_spamcheck_checker ?) that will pass to spamc (I guess we need to make configurable to host for spamc to connect) content that is in the queue, starting from anonymous post. Depending on the spamscore, it will remove the temporary +5 and will had itself additional points (filling also the trackers_spamscore table, unlikely with temporary spamscore; otherwise, the added score would be lost next time it is flagged)
  • a script (sv_spamcheck_monitor) will check every 10 minutes the queue. If it found items that are in the queue for more than, say 20 minutes, then it will remove their temporary spamscore and remove them from the queue, assuming that the server cannot handle the charge and need some air.
  • a script (sv_spamcheck_learn) will check on a regular basis this list of caught spams (spamscore > 4), then it will learn for the bayesian checks. It will store a timestamp in /var/cache/savane so it wont recheck the same marked spam, unless /var/cache/savane is missing (what would be the case if it was moved to another box)