Talk:E-mail

Need to organize this into formal documentation now that I have a better idea of what's going on --Steve 16:27, 5 July 2007 (EDT)

Random notes:

MDA

Mail Delivery Agent

MTA

Mail Transfer Agent

This file README.postfix is part of the amavisd-new distribution,
which can be found at http://www.ijs.si/software/amavisd/

Author: Mark Martinec <Mark.Martinec@ijs.si>
Last updated: 2006-09-15


How to use amavisd-new with Postfix
***********************************

Sections labeled 'COMMENT' may be skipped on first reading.

Your Postfix must not be ancient, it must support parameter 'content_filter'.
Check for the purpose of this parameter in ./README_FILES/FILTER_README
of the Postfix distribution. This file was revised in postfix-1.1.9-20020512,
and again in postfix 20030120, you may want to read the latest version.
In the more recent Postfix documentation the setup described here is known
as 'Postfix After-Queue Content Filter', section 'Advanced content filter'.

For compatibility with previous versions of amavisd the choice of default
tcp port numbers is 10024 and 10025, in contrast to 10025 and 10026 as used
in FILTER_README examples. The service name chosen here is 'smtp-amavis'
instead of 'scan' as in the Postfix documentation.

We are assuming that Postfix is already installed, configured and is
working as expected. As a safety net during experimenting one might feel
better by setting 'soft_bounce=yes' in /etc/postfix/main.cf, and doing
a 'postfix reload'. It will turn hard errors experienced by Postfix into
temporary failures, causing failed mail operations to be retried later.
Don't forget to remove it later when things appear to be running well.


1. Install and start amavisd (as explained in INSTALL - just the daemon,
no helper programs amavis(.c) or amavisd-milter(.c) are needed)

For the first time it is best to start amavisd daemon interactively
and keep it attached to the terminal:

     $ /usr/local/sbin/amavisd debug

From another window check that it is listening on a
local SMTP port 10024 (the default port):

-->  $ telnet 127.0.0.1 10024
     Trying 127.0.0.1...
     Connected to 127.0.0.1.
     Escape character is '^]'.

     220 [127.0.0.1] ESMTP amavisd-new service ready

-->  quit

     221 Bye
     Connection closed by foreign host.


2. With a text editor add to the Postfix master.cf file
the following two entries, e.g. near the end of the file:

smtp-amavis unix -      -       y/n     -       2  smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20

127.0.0.1:10025 inet n  -       y/n     -       -  smtpd
    -o content_filter=
    -o smtpd_restriction_classes=
    -o smtpd_delay_reject=no
    -o smtpd_client_restrictions=permit_mynetworks,reject
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=reject_unauth_pipelining
    -o smtpd_end_of_data_restrictions=
    -o mynetworks=127.0.0.0/8
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000
    -o smtpd_client_connection_count_limit=0
    -o smtpd_client_connection_rate_limit=0
    -o smtpd_milters=
    -o local_header_rewrite_clients=
    -o local_recipient_maps=
    -o relay_recipient_maps=
    -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

Change the 'y/n' to either 'y' or 'n', depending on how you prefer
the smtp and smtpd postfix services to run - either chroot-ed, or not.
See your other (normal) smtp and smtpd postfix services in master.cf
and use the same setting here.

COMMENTS:
- Of all the options specified above in the second entry,
  the one that is essential is the '-o content_filter=' .
- The '-o smtp_send_xforward_command=yes'
  (or '-o lmtp_send_xforward_command=yes' if using LMTP)
  is optional, but recommended - amavisd-new benefits from it since V2.0.
  It does not hurt if specified even if not yet supported by the currently
  running Postfix or amavisd-new.
- the '-o max_use=20' is optional, it overrides the default value of 100,
  and is primarily useful with lmtp, as the Postfix lmtp client is more
  aggressive in keeping the connection open than the smtp client;
- If there is an entry like 'vscan unix - n n - 2 pipe user=vscan ...'
  from an ancient amavisd installation, it is not needed any longer
  and may be removed. Keeping it does no harm.
- for IPv6 enabled MTA, consider: -o mynetworks=127.0.0.0/8,[::1]/128


3. Do a 'postfix reload', check its log file for any complaints,
   and test if it is listening on port 10025:

-->  $ telnet 127.0.0.1 10025
     Trying 127.0.0.1...
     Connected to 127.0.0.1.
     Escape character is '^]'.
     220 yourhost.example.com ESMTP Postfix
-->  quit
     221 Bye
     Connection closed by foreign host.


4. If you want, simulate a mail sent to amavisd and see if it gets delivered
   via Postfix to its recipient. Try first with a simple and clean message,
   then a message with an EICAR test virus pattern which should be recognized
   by all virus scanners (unless all scanners are disabled or not installed):

-->  $ telnet 127.0.0.1 10024
     Trying 127.0.0.1...
     Connected to 127.0.0.1.
     Escape character is '^]'.
     220 [127.0.0.1] ESMTP amavisd-new service ready
-->  MAIL FROM:<test@example.com>
     250 2.1.0 Sender test@example.com OK
-->  RCPT TO:<postmaster>
     250 2.1.5 Recipient postmaster OK
-->  DATA
     354 End data with <CR><LF>.<CR><LF>
-->  Subject: test1
-->
-->  test1
-->  .

***  250 2.6.0 Ok, id=31859-01, from MTA: 250 Ok: queued as 90B7F16F

-->  MAIL FROM:<test@example.com>
     250 2.1.0 Sender test@example.com OK
-->  RCPT TO:<postmaster>
     250 2.1.5 Recipient postmaster OK
-->  DATA
     354 End data with <CR><LF>.<CR><LF>
-->  Subject: test2 - virus test pattern
-->
-->  X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
-->  .

you should get one of the following replies (or similar), depending on
the $final_virus_destiny and *virus_lovers* settings in amavisd.conf:
***  550 5.7.1 Message content rejected, id=16968-01 - VIRUS: EICAR-AV-Test
***  250 2.5.0 Ok, but 1 BOUNCE
***  250 2.7.1 Ok, discarded, id=16984-01 - VIRUS: EICAR-AV-Test
***  250 2.6.0 Ok, id=17041-01, from MTA: 250 Ok: queued as 3F1841A5F5

-->  QUIT
     221 2.0.0 [127.0.0.1] (amavisd) closing transmission channel
     Connection closed by foreign host.

You may need/want to use different sender and recipient addresses.
The test pattern must be entered exactly to be recognized, starting
at the beginning of a line (without indentation).

Depending on the settings in amavisd.conf, the sender (test@example.com)
and the virus administrator may have been sent a (non-)delivery status
notification, the second message should have been quarantined, and the first
message must have been successfully delivered to the recipient. See the log
that is scrolling on the terminal (as set up at step 1) and check for possible
problems.


5. Tell Postfix to start forwarding all mail it receives to amavisd-new
   for content inspection.

To the Postfix main.cf file add a line:

  content_filter=smtp-amavis:[127.0.0.1]:10024

either with a text editor, or preferably using a shell command:
  # postconf -e 'content_filter=smtp-amavis:[127.0.0.1]:10024'

COMMENT:
  The global setting of 'content_filter' in main.cf affects any Postfix
  input service (i.e. smtpd and pickup). If a more selective approach
  is required, the option
    -o content_filter=smtp-amavis:[127.0.0.1]:10024
  may be given in master.cf to selected services only, or the option:
    -o content_filter=
  may override (i.e. clear) the global setting on selected services.


6. Do a 'postfix reload' and watch the logs - both the Postfix logs,
and the amavisd log file (on the screen or wherever you have it directed).

If you get in trouble, you only need to undo the step 5 and do a
'postfix reload'. New mail will no longer be tagged with content filter
routing.

COMMENT:
  The messages that have been received while 'content_filter' was set,
  will still try to get delivered to your old setting of content_filter,
  and will wait in the queue until successful or deleted or expired - or
  until you do:  postsuper -r ALL;  postfix reload

If all is fine, you may abort (^C) the process running 'amavisd debug',
and start amavisd without a 'debug' option, making it detach and daemonize.
There is no need to stop or restart Postfix.

This completes the integration of amavisd and Postfix.
It uses the SMTP (or LMTP) protocol for Postfix->amavisd,
and uses SMTP protocol for amavisd->Postfix communication.
This is the fastest and recommended method, and simplest to set up.


TUNING:

The most important tuning knob is the number of concurrent content filtering
processes allowed. Too low a value does not fully utilize the host resources,
a somewhat high value wastes memory and gains no benefit to the aggregate
mail throughput, while a too high value causes system thrashing and the
total system mail throughput starts to drop. A useful starting value is 2,
a commonly useful range is perhaps up to 10 (or perhaps 20 on hosts with
1 GB of RAM or more, and SA with network tests such as Razor enabled),
but the exact value largely depends on host capabilities and the anti-virus
and anti-spam options in use.

It is imperative that both the Postfix and the amavisd-new use the same value.
Actually the amavisd setting may be higher that the Postfix, but this serves
no useful purpose and just wastes resources. The amavisd.conf parameter is
the $max_servers, the Postfix parameter is the maxproc field in the
'smtp-amavis' entry (file master.cf).

Instead of adjusting the maxproc field of the 'smtp-amavis' service,
one may prefer to leave it a the default '-', and use a main.cf option
for the same purpose:
  smtp-amavis_destination_concurrency_limit = 2

For other tuning hints, see README.performance and:
  http://www.ijs.si/software/amavisd/amavisd-new-magdeburg-20050519.pdf


TO DO 'VIRTUAL ALIAS' MAPPING AND OTHER POSTFIX CLEANUP PROCESSING
BEFORE OR AFTER CONTENT FILTERING?

In a post-queue content filtering setup (a normal amavisd-new setup with
Postfix), a mail message passes through smtpd and cleanup Postfix services
twice, once before the content filter, and the second time when approved
message is passed from the content filter back to MTA. Any transformations
and checks done by a cleanup service are thus performed twice. In simpler
setups this does not matter much, but in more demanding situations one
needs to consider which cleanup instance should perform which task.
See cleanup(8) man page.

In particular, the following should be considered:

- masquerading

- canonical address transformations
    placed before the content filter:
      content filter will see canonicalized envelope addresses
      (e.g. external addresses)
    placed after the content filter:
      content filter will see largely unmodified envelope addresses

- virtual alias transformations of envelope recipient addresses
    placed before the content filter:
      content filter will see modified (e.g. internalized) envelope addresses
    placed after the content filter:
      content filter will see largely unmodified envelope addresses

- built-in content checks like the header_checks, body_checks, mime processing
    placed before the content filter:
      the usual placement, checks should be performed as early as convenient
    placed after the content filter:
      most built-in content checks should not be performed again to save time
      and prevent late bounces. An exception may be the 'placing on hold'
      of a mail message that the content filter considered a potential threat
      and inserted a header field 'X-Amavis-Hold: reason', which needs to be
      done after content filtering.

- automatic BCC recipient controls
     should only be done once to prevent mail duplication. The same
     applies when virtual mapping is used a "poor man's" mailing lists.
     Adding recipients is normally placed after content filtering;

- resource and rate controls
     should be done before the content filtering, and should be disabled
     or be more liberal in the cleanup service after the content filter;

To exercise full control over which cleanup service will perform which
e-mail address mapping (virtual alias, canonical, masquerading), and
which (if any) header/body checks, one needs to use two cleanup services:

- add a new service 'pre-cleanup';
- (optionally) add options to existing service 'cleanup';
- add option 'cleanup_service_name=pre-cleanup' to existing services
  'smtp' and 'pickup';

as described further down.

If the full flexibility of having two cleanup services is not needed
and Postfix is snapshot 2.0.13-20030706 or later, there is a new parameter
'receive_override_options' which eliminates the need for two cleanup
services in some more straightforward cases (not all features of having
two cleanup services are available). The idea is to use:
  -o receive_override_options=no_address_mappings
for main incoming services (like smtpd and pickup), and the:
  -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks
for the post-content-filter smtpd service on port 10025.
See smtpd(8) man page and the FILTER_README and ADDRESS_REWRITING_README
files in the Postfix documentation directory README_FILES.
The receive_override_options=no_address_mappings also avoids the need
for moving always_bcc option from main.cf to master.cf in common cases.

--Steve 14:29, 5 July 2007 (EDT)