Spamassassin and qmail

Discussion in 'PSA 2.5 HOWTO' started by jimroe, May 19, 2002.

  1. jimroe

    jimroe Guru

    Messages:
    2,067
    EDIT
    As this thread expands into it's 8th page it's taking on a life of its own - which is good. But, many who are starting the installation now are doing so without reading the whole thread, and running into problems that were already solved. This is not the easiest install to do. It involves a lot of programs, a lot of configuration, and attention to detail, but it's not difficult unless you miss some critical step. So, do yourself a favor, and read ALL of this post before you start, make notes on key discoveries, and if you have problems, come back here and read this thread again. Learn from all the others who have done this and don't re-invent the wheel.


    This is driving me nuts!

    I have installed Spamassassin on my Plesk 2.5.1 box (in conjunction with rblsmtpd and Odeiavir - I really hate viruses and spam), and I'm struggling with getting it to add the score and spam detection messages to the email.

    Following what was suggested here (and worked before) for Odeiavir, I created this .qmail file (I'm running the daemon "spamd" and the client "spamc" but the same problem happens if I try the perl script):

    | /usr/bin/odeiavir
    | spamc -f
    | true
    ./Maildir/

    This passes email correctly, and spamc logs the Spamassassin score and scanning results to /var/log/maillog, but it doesn't add the information to the email. But, if I pass a known spam test email through Spamassassin (cat sample-spam.txt | spamc), spamc outputs a modified email with the score and warning to stdout as I would expect, so it is working. I figure this is some subtle issue with the .qmail file but I can't figure it out. I even took a message that I sent myself (which didn't get the Spamassassin report added to it) and passed it through spamc and these lines were correctly added (and should have been present in the email as delivered):

    X-Spam-Status: No, hits=0.0 required=5.0 tests= version=2.20
    X-Spam-Level:

    Does anybody have this working that could share the solution with me??

    Please??
     
  2. jimroe

    jimroe Guru

    Messages:
    2,067
    I feel stupid answering my own posts, but

    I see no one took this - well, after much digging, interpreting, and frustration, here's the answer:

    spamc (and the perl script spamassassin also) take their input from a file, but output to stdout (ie the terminal). The problem lies in that qmail passes mail through as files. So, while it's easy for qmail to pass a message to spamassassin as directed in the .qmail file, there's no way for spamassassin to pass it back as a file to qmail for delivery. I found a number of (complicated) solutions to this, but finally landed on one simple one. It requires one program to be installed - safecat (http://www.nb.net/~lbudney/linux/software/safecat.html). Safecat will take data from the terminal, and forward it to a .Maildir structure - it's written by Dan Bernstein (who wrote qmail) so it integrates well. Once you install the program (RPM or tarball) you pass the output from spamc to it and let safecat write the file to .Maildir. Here's what the .qmail file looks like:

    | /usr/bin/odeiavir
    | true
    | spamc -f -u root | /usr/local/bin/maildir ./Maildir
    #./Maildir/

    I left the last line of the original in to show what it used to be like.

    I now get all email with the spamassassin headers and messages added so I can monitor (for a while), fine tune the rules, and then automatically drop them or divert them to spam mailboxes.
     
  3. Dan S.

    Dan S. Guest

    Very nice!
    I tried to get spamassassin working with procmail, and I gave up after messing with it for a few hours. This method is much cleaner. I'll have to test it out when I get a chance :)
     
  4. jimroe

    jimroe Guru

    Messages:
    2,067
    Followup

    Spamassassin is easier to get working with procmail than with qmail because you don't have to convert the output back to a file, but since Plesk, by default, uses qmail all the way, you have to work to get procmail integrated. And, qmail is safer because it writes everything to disk during transfers, so nothing gets lost in a transfer. Even with all my problems, I never lost any emails or test messages, even though a lot got deferred. Safecat is coded the same way qmail is (it actually dumps the input to a temporary file while it processes just in case), so it keeps a robust string through the chain. The biggest problem is the lack of documentation on most of this, so it took some time.

    I just noticed in the previous post - I used

    /usr/local/bin/maildir

    as I processed .qmail even though I was saying I used safecat to process the message. "maildir" is a shell script that installs when you compile safecat: "maildir" invokes safecat in the following line from extracted from "maildir":

    /usr/local/bin/safecat "$1"/tmp "$1"/new


    Since the previous post, I fine tuned a shell script I found that looks at the spamassassin report after processing the message, and diverts spam to a spam drop address (it could just as easily > /dev/null), but forwards non spam messages directly to your ./Maildir. (needs one additional program to be installed)

    This whole thing integrates smoothly with Plesk & qmail, as well as the Odeiavir scanning approach documented in another thread in this forum. Together, you have a smooth working approach to protecting your clients from both viruses and spam - at their choice. A nice package.

    If there seems to be some interest in this combination, I'll do a more complete "how-to" post so others can benefit.
     
  5. DigitalXWeb

    DigitalXWeb Guest

    You have my interest in a combination "how-to" :D Nice job..
     
  6. jimroe

    jimroe Guru

    Messages:
    2,067
    How To

    Look for a combination "how-to"- which will integrate virus scanning, rbl scanning before the message enters the delivery chain, and spamassassin, in 7 to 10 days (other committments). I already have a script modified (but not fully tested) which can be configured (by the admin) to deliver non-spam messages with, or without, the SA header by simply setting a flag in .qmail. I hope to be able to also deliver a spam message to a spam drop, with or without the SA header data (right now it always has it) - this is what I'm working on now, so no promises.

    I also would like a cl script to install Spamassasin for specific domains / users without manually editing .qmail files, although I don't anticipate this will be automatic for all users - I think the admin needs control.

    I have a pretty complete README written, but it will change slightly as the other mods get implemented. The whole package will be too lengthy for a post, but will be avaliable as a tarball with all scripts, instructions, and links to download all the component programs included. I'll post a link to it in this thread when it's ready.
     
  7. jnoble

    jnoble Guest

    I am also greatly interested in teh how-to on this...

    Eagerly awaiting Instruction from one who knows

    James
     
  8. Peter Enoch

    Peter Enoch Guest

    Question about SpamAssassin!

    I have tried to install SpamAssassin and it seems to work together with Plesk just fine.

    But I have one problem I hope someone can help me with. As default I get the e-mail, just saying that this is probably a SPAM e-mail.

    Can't I have SpamAssassin send the SPAM e-mail it finds to a specific e-mail adress?

    Regards

    Peter Enoch
     
  9. jimroe

    jimroe Guru

    Messages:
    2,067
    Yes you can

    You can configure spamassassin so that it sends the entire email to any email address you wish; with my current script, the email will also include the headers and other data added by spamassassin as well as the complete email message - I am working on how to get it to just send the original email without the spamassassin data as an option, but I don't have that detail working yet.

    I have a little more to do with the documentation before I post the whole set of scripts to do this - a few days yet.
     
  10. jimroe

    jimroe Guru

    Messages:
    2,067
    Composite install

    Rather than wait any more, I've decided to post what I have in its current state. Highlights:

    1. Implementation of Odeiavir scanning with F-Prot virus engine - incoming emails only.
    2. Implementation of Spamassassin scanning. The scanning script enables forwarding of spam email to any email address, and a toggle to include or exclude the spamassassin header data on all other email messages.
    3. A README file that explains all the installation steps as well as links to all the programs needed for installation. All scripts needed are included.

    All this is packaged in one tarball, available at:

    http://www.personalcopy.net/plesk-nospam-1.0.tar.gz
     
  11. Dan S.

    Dan S. Guest

    Looking good.
    I'll give it a test run on a clean test machine as soon as I get a chance.
     
  12. DigitalXWeb

    DigitalXWeb Guest

    Re: Composite install

    Question about this, I am in the process of installing this on a test machine and after going through part of the install and reading the README's in each of the packages, in the odeiavir README it states to add the following to the top of your users .qmail files :
    |/usr/bin/odeiavir

    That is fine for the current email accounts on the system, but is this stuff added automatically to any new ones created or do we have to manually go in and do the same?
     
  13. jimroe

    jimroe Guru

    Messages:
    2,067
    When you install the "insert_odeiavir.cgi" script and add it to the crontab, it will run every hour - when it runs, it will create a backup .qmail file, and add the required line to each .qmail file it finds - you don't have to do it manually. Even if you add new email accounts, they will be updated in at most an hour. If you are really paranoid, you could manually run the script.

    The changes required for spamassassin must be done manually, however - I haven't gotten around to writing a script to do that - and I'm not sure I even want to - not all users would necessarily want spamassassin on their email.
     
  14. DigitalXWeb

    DigitalXWeb Guest

    Thanks for the info and the how-to!!
     
  15. awlane

    awlane Mega Poster

    Messages:
    157
    Thanks jimroe,

    This is awesome. Good documentation is priceless.

    I got everything to work on RH7.2, PS 2.5.3 first try.

    I am not using the Spam detection bit anymore because I'd rather modify the SUBJECT line of detected spam and deliver it as normal rather than sending all spam to a specific dropbox. If anyone has any pointers on how to modify the script to accomplish this, don't hold back! Post it right here. You know you want to! :p
     
  16. jimroe

    jimroe Guru

    Messages:
    2,067
    Should be easy

    If you want the Spamassassin-modified message to be delivered to the "normal" mailbox user instead of a special mailbox for Spam, you need to change the areas of the script which do the redirection, and tell the script to deliver to the normal user. Look for each place in the script where you see the line:

    echo "$output" | $INJECT -a -f"$SENDER" $FORWARD

    and replace it with:

    echo "$output" | /usr/local/bin/maildir ./Maildir

    Example:

    Original:
    <snip>
    if [ -z "$SHOWSAH" ]; then
    # inject the original message
    #echo "$input" | $INJECT -a -f"$SENDER" $FORWARD
    echo "$output" | $INJECT -a -f"$SENDER" $FORWARD
    if [ $? -eq 0 ]; then
    # so qmail will not do any further deliveries in .qmail file
    exit 99
    fi
    </snip>

    Modified:
    <snip>
    if [ -z "$SHOWSAH" ]; then
    # inject the original message
    #echo "$input" | $INJECT -a -f"$SENDER" $FORWARD
    #echo "$output" | $INJECT -a -f"$SENDER" $FORWARD
    echo "$output" | /usr/local/bin/maildir ./Maildir

    if [ $? -eq 0 ]; then
    # so qmail will not do any further deliveries in .qmail file
    exit 99
    fi
    </snip

    This change will effect ALL users since you're changing the only copy of the script and all users use the same script. If you want some users' mail to be redirected, and others to be passed through but flagged, you could make a copy of the original script and modify the copy as noted above. Then, invoke the correct script in the user's .qmail file depending on what the user wants to have happen.

    Since Plesk allows unlimited email users even on a limited-domain license, there's no reason why you couldn't set up a Spam-drop email address for each user, which is why I didn't provide a toggle in the script.

    EDIT:

    Just an additional note - I haven't tested the above, and there ARE other ways to do this, so if this doesn't work let me know and I will test and come up with a solution that does work.
     
  17. awlane

    awlane Mega Poster

    Messages:
    157
    Cool, I will try it this weekend.
    I am already very pleased with the Antivirus component. It;s interesting to see how many viruses people get every day.
    In 3 days of running ODEIAVIR I have had at least 5 unique incidents.
     
  18. awlane

    awlane Mega Poster

    Messages:
    157
    Stuff the weekend. I am doing it now.

    Already running into trouble. Following all instructions but when I run ntsysv, spamd is not listed.

    I did chkconfig spamd from within /usr/local/bin, where spamd resides but it doesn't add the service.
     
  19. awlane

    awlane Mega Poster

    Messages:
    157
    K,

    spamd is now running and starting on boot, ODEIAVIR is still working globally but spam mail does not arrive any longer.


    This happens on locally injected spam and on external spam e-mail sent from yahoo.com.

    The user's .qmail file in /usr/local/psa/qmail/mailnames/domain.com/abuse/ is

    | /usr/bin/odeiavir
    | true
    | /usr/bin/ifspamh abuse@domain.com 1
    ./Maildir/

    Basically any spam mail now gets rejected by QMAIL after about 10 minutes because it "looped".

    <abuse@domain.com>:
    This message is looping: it already has my Delivered-To line. (#5.4.6)

    If I put | /usr/bin/ifspamh somebody@hotmail.com 1 in the .qmail file, normal e-mails go through and spam goes to hotmail.
     
  20. jimroe

    jimroe Guru

    Messages:
    2,067
    Here's the problem

    When you have an email account for domain.com named abuse, and you put an email forward for spam messages in the .qmail file of abuse@domain.com, then qmail tries to forward a message from abuse@domain.com to abuse@domain.com - which certainly loops back to itself, and thus the error message from qmail. Sending Spam to ANY other mail account works fine, as you've discovered.

    If abuse is the mail account where spam mail is supposed to be sent, then you do NOT want to include the line:

    | /usr/bin/ifspamh abuse@domain.com 1

    in the .qmail file for abuse. Also, abuse shouldn't really be a "published" email address in that you don't want normal email going to a spam drop account - only spam as determined by Spamassassin.

    If indeed you do want Spamassassin to monitor email sent to abuse@domain.com, but send any spam it finds to abuse@domain.com, then you need to make the modifications I detailed in my earlier post to the ifspamh file. I'm not quite sure what would happen here though - you might get multiple Spam headers added because mail from other accounts directed to abuse would already have Spam headers, and then they would get processed again and might get a second set added - but at least it won't loop.

    Effcetively what those changes do is to tell ifspamh to deliver spam to the normal ./Maildir for that user instead of forwarding it to a different account.

    You didn't mention how you fixed the problem with spamd not showing up in ntsysv, but for the benefit of the thread - in order for a program to show up in ntsysv OR chkconfig --list, it:

    1. Must be located in /etc/rc.d/init.d
    2. Must have certain descriptions in its header - even though they appear to be commented out. Specifically, the lines:

    # chkconfig: 345 80 30
    #
    # description: spamd is a daemon process which uses SpamAssassin to check
    # email messages for SPAM. It is normally called by spamc
    # from a MDA.

    must be present.
     

Share This Page