November 2, 2004 amavisd-new-2.2.0 release notes SECURITY: - modified MIME entity traversal to include MIME container parts (e.g. multipart/*) as pseudo parts. Such pseudo-parts do not carry any body contents but do show up (with their MIME content type only) in the tree structure as seen by banned names checks. This makes it possible to specify more complex banned rules based on the placement of leaf nodes within or outside of mime multipart containers. This also re-enables the possibility to check such MIME wrappers for banned MIME Content-Types (most notably for the message/partial and message/external-body), which was lost with the change of internal representation of mail parts in version 2.0 (amavisd-new-20040701); - preserve original zip archive for virus scanners to see, if the archive contains any zero-length members (Archive::Zip module chooses not to extract members with declared zero size, even if the size does not correspond to the actual size); based on a patch by Dirk Datzert; - add tests to mime_traverse to verify that the files produced by MIME::Parser really do exist and are readable; (and sent a patch adding I/O checks for MIME::Tools to its maintainer; please use MIME::Tools 5.414 or later from CPAN to avoid possibility of full /tmp partition to cause infected or spam mail to pass through); - changed recommendation in INSTALL to choose the latest version of MIME-Tools from CPAN - the 5.415 at the time of this writing; - do_unrar: recognize encrypted entire archives (not just their individual members), and flag mail as undecipherable; - recognize file(1) report/^MS Windows\b.*\bDLL\b/ as short type 'dll'; add 'dll' to example patterns in amavisd.conf and amavisd.conf-sample; add name extension '.cpl' to the list of basic banned names; INCOMPATIBILITY: - incompatible change: the default value for $recipient_delimiter is now undef and no longer '+'; adding address extensions must now be explicitly enabled; - minor change in the default X-Virus-Scanned: header field, see below; OTHER CHANGES SINCE 2.1.2: - rewritten LDAP modules, by Michael Hall; - improvements in handling of double errors (errors that occur while handling consequences of some previous error); be permissive on failures in DESTROY methods; prevent some cases for child process not being able to sign off from the nanny database; - enforce $max_requests more strictly, dropping SMTP session after task count is exceeded by one; this is in response to the new smtp session caching behaviour in Postfix, which is now much more persistent in keeping the session open on a busy mailer; although dropping session at the server side is discouraged by RFC 2821, this change was recommended on the postfix-users mailing list; - added a site-wide mail size limit $smtpd_message_size_limit, and a per-recipient mail size limit lookups @message_size_limit_maps along with SQL and LDAP fields, making it possible to reject mail based on its size. The list of lookup tables maps recipients to mail size in bytes, e.g.: @message_size_limit_maps = ({'user1@example.com' => 20_000_000, 'user2@example.com' => 15*1024*1024, 'user3@example.com' => 0, # uses global limit '.example.com' => 10*1024*1024 }); $smtpd_message_size_limit = 20*1024*1024; # global limit if nonzero A value of 0 or undef disables the check and is a default. A per-recipient limit is bound on the high side by the $smtpd_message_size_limit, and on the low side by 64kB, which is a minimal allowed size limit imposed by RFC 2821. This limiting really belongs to MTA and is only partially supported here (no admin notifications, no quarantine, no final_*_destiny configurability). It is mostly provided here to be able to place some sanity limit on runaway or malicious clients, or if someone insists on using amavisd-new in a pre-queue filtering setup; suggested by Tuomo Soini; - internal: add new object $msginfo->mail_text_fn to hold the file name of the original mail, decouple the location of file email.txt from the temporary directory, which was implied until now. This is presently needed for the Courier interface. Add optional attribute 'mail_file' to the AM.PDP protocol, see README.protocol; - in banned parts descriptor strings which are used in banned name checking, provide a 'T=empty' short type for empty mail parts, including for empty MIME parts (instead of omitting T=... altogether). This can be used in banned rules to test for empty parts, generally or restricted to empty MIME parts; suggested by Ricardo Stella and Stephane Lentz; - a banned lookup result (which is interpreted as boolean for most purposes: zero or empty for false, anything else for true) may give a result 'DISCARD' (which is true as well), which will disable DSN if it turns out the mail was blocked by such banned rule (akin to viruses_that_fake_sender and spam_dsn_cutoff_level). Here is an example rule in $banned_namepath_re: [ qr'(?# BLOCK EMPTY MIME PART APPLICATION/OCTET-STREAM ) ^ (.*\t)? M=application/octet-stream \t(.*\t)* T=empty (\t.*)? $'xmi => 'DISCARD' ], Suggested by Ricardo Stella and Stephane Lentz; - fix 'Insecure dependency in unlink' in sub files_to_scan which could happen when some decoder left non-regular files in the directory; reported by Andrzej Kukula; - bug fix: only insert LDAP and SQL lookups objects into lists of maps at the first task of a child process, later tasks should not insert duplicates; - new subroutine do_ar and new configuration variable $ar to handle standard Unix archives, including Debian binary packages; - recognize a Unix archive and give it a short type .a; - recognize a Unix relocatable binary and give it a short type .o; - do not penalize SMTP status "554 5.1.1 Error (DATA): no valid recipients", the situation arises regularly when pipelining is in effect and all recipients happen to be rejected; - protect spaces and newlines when logging broken Message-ID and Resent-Message-ID header fields (macros %m and %r) to facilitate log parsing; protect newlines in logged Subject header field (macro %j); parsing difficulty pointed out by Chris Lee; The present solution uses =XX (hex) encoding and is a quick and dirty fix. A cleaner solution to avoid double sanitation of special characters is needed, but would involve a deeper reorganization; - updated example list of "banned extensions - long" in amavisd.conf and amavisd.conf-sample; - change the default lock file name from "$MYHOME/amavisd-$$.lock" to "$MYHOME/amavisd.lock", to avoid inventing a new name at every restart and leaving old files behind; pointed out by Dale Walsh and Martin Orr; - updated av entry for nvcc (Norman Virus Control v5 / Linux) to include statuses 10,11, and 2,14 to the status lists according to documentation; password protected or corrupted archive (status 11) was not recognized as non-infected status; thanks to Michael Ramke of Norman Data Defense Systems GmbH; - updated DrWebD entry to recognize and ignore flag DERR_SKIPPED; - support Mail::ClamAV 0.12 and 0.13 or later, which is incompatible with 0.11 due to change of constant names in the underlying ClamAV library; - added 'check-jpeg' example entry to the @av_scanners list and provide the associated module JpegTester.pm; it offers a fully-fledged check for jpeg comment field buffer overflow attempts; should serve mainly as an example for adding similar quick responses to new threats; - added 'check-jpeg-simple' example entry to the @av_scanners list (only in amavisd.conf-sample); it offers a quick check (and not very exact one: checks only the first 32kB) for jpeg comment field buffer overflow attempts; should serve mainly as an example for adding similar quick responses to new threats; - relax too restrictive sanity check on temporary directory name when accepting requests from a helper program or via AM.PDP protocol (e.g. with sendmail milter setup); reported by Babu Kanagala; - relax allowed set of characters when receiving XFORWARD attribute values, it turns out that characters like '=' and '+' are allowed; - when using "bsmtp:" delivery method suppress X-Envelope-From and X-Envelope-To header fields, as the addresses are already available in the envelope; - when using the "bsmtp:" quarantine method the *_quarantine_to was completely ignored, which made it impossible to turn off quarantining selectively for certain users by specifying an empty or undef value. Since 2.2.0 an empty *_quarantine_to turns off quarantine for a recipient regardless of the quarantine method. A nonempty string in *_quarantine_to (the exact value is still ignored) must now be used even with "bsmtp:" to enable quarantining. Inconsistency discovered by Sean Doherty; - suppress leading $QUARANTINEDIR string from the value of macro %q, thus hiding the absolute file path from notifications; - add configuration variable $local_client_bind_address (and equivalent policy bank key), to allow for explicitly binding local socket address to a specific interface in SMTP client; suggested by Wouter de Jong; - keep whatever (if any) file results from gunzip and family (do_uncompress) even if the decompressor's exit status is nonzero; reason: gzip returns status 2 when decompressing file with trailing garbage; reported by Tobias Reckhard; - collect declared original file name from gzipped (and friends) files if reported by file(1), making them available to banned name checking; - avoid unnecessarily checking white/black lists if spam checks will not be used (e.g. infected mail); - use qquote_rfc2821_local to properly quote e-mail addresses in the most visible log entries; - if there is more than a single (or less then one) occurrence of %k in the SQL SELECT clause template, multiplicate the set of query keys accordingly, making possible more complicated custom SELECT clauses; - don't forget to load amavis policy delegation protocol support code if AM.PDP is explicitly requested in $protocol, even if not listening on a Unix socket; - add 'queue_id' attribute to the AM.PDP protocol; equivalent to a change in the Courier support code by Martin Orr; - include the declared (faked) sender address in the virus recipient notification template, in addition to the originator IP address; - add macro %Q and method Amavis::In::Message::queue_id, holding a MTA queue ID if available (in Courier and milter/AM.PDP setup); by Michael Musikhin (through Martin Orr); - add macro %y to show elapsed processing time; suggested by Ed Walker; - sanitize newlines and spaces (and some other characters) when moving syntactically invalid Message-ID and Resent-Message-ID to macros %m and %r for logging purposes; resulting wrapped main log entry reported by CRivera; - bring up syslog priority to LOG_NOTICE when debug or debug_oneshot is in effect; - make a product name, version ID and version date available as separate variables to avoid the need to parse $myversion for the purpose of customizing e.g. the setting of $X_HEADER_LINE; based on suggestion by Dale Walsh; the re-introduction of a date ($myversion_date) also suggested by Stephane Lentz. Added variables: $myproduct_name, $myversion_id, $myversion_id_numeric, and $myversion_date, which serve to assemble the $myversion. Modified the default templates of $smtpd_greeting_banner and $smtpd_quit_banner to take advantage of the new variables. Changed $X_HEADER_LINE default from "by amavisd-new at $mydomain" to "$myproduct_name at $mydomain" and added an example of a $X_HEADER_LINE setting with version number included to the amavisd.conf-sample; - added SQL fields 'virus_admin' and 'spam_admin' to lookup lists @virus_admin_maps and @spam_admin_maps; patch by Robert Collier; - add a log message 'SPAM-KILL, ...' (at log level 3) for not-passed mail, to complement the 'SPAM-TAG' log message for passed mail; - add Mail::SpamAssassin::Plugin::SPF to a list of modules that SA fails to load at init time; - prevent sending the same SMTP response more than once, if the first attempt failed due to disconnected SMTP session; - fix a double-@ formatting buglet in the log message "adding address extension _spam to user@@domain", reported by Vincenzo; - add kill('TERM',$$) to the arsenal of attempts to get rid of unwanted forked process; - includes rpm spec file, including the init script, contributed by Marius Andreiana, based on previous work by Dag Wieers; - document the localization template directory contents (in file amavisd.conf-sample) when read_l10n_templates is used; thanks to Joël Brogniart; - includes file Macintosh.tar.gz, which contains auto-startup scripts and installation instructions for Mac OS X, contributed by Dale Walsh of the Dale Enterprise L.L.C. --------------------------------------------------------------------------- September 6, 2004 amavisd-new-2.1.2 release notes - fixed (hard)black- and white-listing on static lookup tables which failed to match any sender; reported by Derck Floor; - use $hdrfrom_notify_recip address in the From: field for recipient notifications, instead of $hdrfrom_notify_admin; inconsistency pointed out by Ekkehard Burkon; - the 'neutral' sender notification template was joining the Subject and the Message-ID header fields into one longer Subject when it was reporting some nondelivery other than the 'invalid characters in header'. Likewise the first body line of this same DSN was eaten up: "This nondelivery report was generated by the amavisd-new program" (the problem was introduced in amavisd-new-20030616 and never reported); - in amavisd-agent, amavisd-nanny, amavisd: extend the signal and error handling in code sections holding bdb locks from just ignoring the SIGINT, to controlled catching and re-signaling several signals and error conditions; problem reported by Tom Mulder; - suppress duplicate names from the list of virus names in macro %V; by Gregor Weiss; - fine-adjusted log level of some log messages; - discard leading and trailing whitespace from the macro %t (Received trace); - extend the search for IP in the Received trace from 4 to a maximum of 6 entries; - ignore private IPv6 addresses (RFC 3513: link-local, site-local, multicast) when searching through Received trace for the origin of mail; - place mail header field X-Envelope-From in front of the X-Envelope-To in quarantined mail; also changed case of X-Quarantine-id into X-Quarantine-Id for consistency with other header fields; - provide new macro %e which evaluates to our best guess of the originator IP address collected from the Received trace, complementing similar macros %t, %a and %g; suggested by Gregor Weiss; - add the result of macro %e to the default 0-level log entry; - provide new macros %u and %U to evaluate to a timestamp of the message reception similar to an existing macro %d (RFC 2822 local date-time); the (%u) as Unix time (seconds since 1970-01-01T00:00Z as a decimal integer, suggested by Gregor Weiss), and (%U) as ISO 8601 (EN 28601) UTC date-time; - avoid some empty lines in default DSN templates and fix some inconsistencies in their formatting; - internal: collect existing common code for time formatting as new subroutines iso8601_timestamp and iso8601_utc_timestamp; collect existing common code to find IP address in the Received trace as a new subroutine best_try_originator_ip; - bump up the version number in $myversion - the 2.1.1 still presented itself as 2.1.0; - add a note about a data structure difference between @score_sender_maps and $per_recip_blacklist_sender_lookup_tables (amavisd.conf-sample, amavisd) --------------------------------------------------------------------------- August 24, 2004 amavisd-new-2.1.1 (amavisd-new-20040824) release notes - unconditionally initialize @banned_filename_maps to (\$banned_filename_re), otherwise $banned_filename_re is ignored by default (unless amavisd.conf explicitly assigns to @banned_filename_maps); a patch by Thomas Jarosch; - fixed inappropriate log entry in SQL whitelisting: wbl: (SQL) recip whitelisted sender <...>, unexpected wb field value: "1"; reported by Carlos Horowicz; - added missing import of &ca to the amavisd-new-courier.patch; by Martin Orr; - produce a warning when there is an unknown field in the policy bank to be loaded; - with delivery method 'bsmtp:' prepend a directory $QUARANTINEDIR to the file path if not explicitly specified, to behave like the 'local:' delivery method, making it possible to hide full path from the X-Quarantine-Id and notifications; a patch by Thomas Jarosch; - pre-load SA 3.0.0 module Mail::SpamAssassin::Plugin::Hashcash to make it available in the chroot jail; - pre-load modules Mail::SpamAssassin::SpamCopURI and URI::* for SA older than 3.0.0; - enhancement to amavisd-nanny: when terminating a process and SIGTERM produces no result for some time, try SIGKILL; contributed by Philip Engdahl; --------------------------------------------------------------------------- August 15, 2004 amavisd-new-2.1.0 (amavisd-new-20040815) release notes The 2.1.0 release is mostly a maintenance release over 2.0, with only a handful of smaller features added. Based on a manual code audit the number of smaller internal code changes is rather extensive, some changes dealing with long-standing known deficiencies, minor bugs, documentation problems and typos. Only a few fixes are for new bugs introduced in 2.0. The files amavisd.conf, amavisd.conf-default and amavisd.conf-sample have been extensively reworked, with the hope to suit better the new installations, while possibly causing some head-scratching for existing installations when looking at a diff. The file amavisd.conf is the one that should serve as a sound base for the initial config file, while keeping an eye on the list of all variables and their defaults in amavisd.conf-default. The amavisd.conf-sample is being phased out of active maintenance, and should serve mostly as a set of examples and the source of documentation until better documentation is available. Two nice features are available: - the use of BerkeleyDB and libdb is now optional; controlled by variables $enable_db and $enable_global_cache; - a new program 'amavisd-nanny', with the accompanying instrumentation in amavisd, displays the general health of all amavisd child processes, reports crashed ones and attempts to kill long overdue processes; It is still experimental and minimalistic, problem reporting is currently only to stdout. Other changes: - incompatible change since 2.0: the use of BerkeleyDB is now off by default; The use of BerkeleyDB and libdb is made optional, controlled by variables $enable_db and $enable_global_cache, both false by default. $enable_db: enables the use of BerkeleyDB/libdb (for SNMP counters database and nanny, and optionally for cache); $enable_global_cache: enables the use of libdb-based cache when $enable_db is also true; If either the $enable_global_cache or $enable_db are false, cache of mail body MD5 digests is kept in child-local memory as in pre-2.0 versions, and is therefore local and short-lived, with lower expected cache hit rate; - incompatible change: DSPAM 3.0 or better is needed (if $dspam is enabled), no longer works with 2.x; - incompatible change: changed name of the (hardly ever needed) configuration variable auth_supported_out to auth_required_out, to better reflect its semantics (should be true if MTA server to which amavisd is sending notifications and forwarding mail requires authentication (AUTH smtp command)); - a new small program included: amavisd-nanny is a program to show the status and keep an eye on the health of amavisd child processes (experimental); - fixed a bug in lookup_acl where a "." did not act as a catchall; thanks to JP; - fixed a problem in SQL lookup which could return undef even when not all the matched records had NULL in the field; - fixed compatibility with old 'amavis' helper program ('delivery_care_of' defaulted to 'client', instead of depending on the presence of ldaargs), reported by Charlie Schluting and Christer; - fixed long standing problem in do_ascii, which could return without calling Convert::UUlib::CleanUp, occasionally spilling state into subsequent mail checks within the same process; - fixed macros %D, %O and %N when log_recip_templ is being expanded; a patch by Ed Walker; - fixed recognition of separators in a nested call during macro expansion; - pre-load missing modules Net::LDAP, Net::LDAP::Schema, Net::LDAP::Search, and Net::LDAP::RootDSE; suggested by Paul Jacobson; - fix locale-related bug in rfc2822 date generation, where we were restoring the saved LC_TIME value to LC_CTYPE (!); a patch by Henrique de Moraes Holschuh / Debian support team; - protect from signals while bdb cursor holds a lock; - new subroutine inherit_header_edits() and slight code restructuring makes possible for spam_scan() and other code before the final delivery to start submitting common header edits into $msginfo->header_edits, avoiding the need for passing them through global variables; - now a loglevel-based automatic syslog priority assignment can no longer lower a message syslog priority below the syslog priority specified in the $SYSLOG_LEVEL, it can only increase it; the violation of the least-surprise principle pointed out by Andy Dills; - a small optimization in logging: a new subroutine ll allows to save time in preparing complex log entries when we know their log level exceeds the current log level and won't be logged; - in default macro templates $log_templ and $log_recip_templ: * placed 'spam' condition before 'bad header' for consistency with program behavior; * added reporting of tag/tag2/kill levels in $log_recip_templ (experimental: macro names may change in future versions); * rewritten templates using negation (i.e. [?x|1] ) to avoid unsightly selector nesting; - MIME decoding is now allowed to exceed the decoding quota, avoiding the problem when a small quota settings might not allow even a plain mail through; - override LC_TIME to "C" on every log message, to work around issues with Unix::Syslog, which would log stuff with the date stamp localized, which syslog would dislike and add its own, and the resulting mess is not recognized by amavis log processors; a patch by Henrique de Moraes Holschuh / Debian support; - changed dspam command line options to work with dspam 3.0 (no longer with 2.x), a patch by Ron Ohmer, Nabil Sefrioui, and Reech; - dspam header fields are now inserted into passed mail if all recipients are local; - supply better defaults for $hdrfrom_notify_sender, $hdrfrom_notify_recip, $hdrfrom_notify_admin and $hdrfrom_notify_spamadmin, similar to defaults from amavisd-new-20030616-p10; - when parsing output from the 'AVG grisoft' virus scanner, don't include CR in virus name; reported by Vernon A. Fort; - new file 'amavisd-new-qmqpqq.patch' provided by Martin Solciansky, (similar to fixes by Christopher Odenbach) making it work again with the current version of amavisd-new; - use lstat instead of stat, and test for soft links wherever appropriate; - remember inode and device number when creating temporary directory and temporary file, and test for change before removing them; - enhanced security: certain tainted values are allowed to enter deeper into program, untainting them only where and when really necessary; - avoid a taint problem in Mail::ClamAV; - added AV entry for CAI eTrust Antivirus; by Stephane Lentz (requires a suid shell wrapper around inocmd32); - added status 9 to the set of infected statuses for the drweb command line scanner (DrWeb Antivirus); - use our subroutine q_encode to Q-encode header fields from the notification templates, instead of MIME::Words::encode_mimeword (a similar fix in 2.0 applied only to encoding of modified headers in passing mail); - add attribute 'x-spam-type=original' to the Content-Type header field (the SpamAssassin's code to recognize an original email) when defanging spam, facilitating reporting spam via SA to other spam fighting tools; a patch by Brian May; - add a note to amavisd.conf that $sa_auto_whitelist has no effect on SA since 3.0.0 - SA has now a configuration file option 'use_auto_whitelist'; - turn off timer in post_process_request_hook() to avoid periodically recreating child processes on an idle machine; - added @mynetworks_maps and enhanced lookup_ip_acl() to take a list of lookup tables: currently members can be an array ref (as before), or a hash ref (new) or a plain constant (new); - generalized @debug_sender_acl into @debug_sender_maps along the lines of other lookup tables, and make it part of policy banks; - added @warnvirusrecip_maps, @warnbannedrecip_maps, @warnbadhrecip_maps; - added @spam_subject_tag_maps and @spam_subject_tag2_maps, to allow per-recipient spam tags string; suggested by Ed Walker; Note, there is an inconsistency in names of legacy variables and the new @*_maps, in an attempt to rectify an unfortunate choice of name for seldomly used variable $sa_spam_subject_tag1: @spam_subject_tag_maps = (\$sa_spam_subject_tag1); # exotic @spam_subject_tag2_maps = (\$sa_spam_subject_tag); # in common use - added configuration variable $auth_reauthenticate_forwarded, which directs amavisd to apply its own credentials ($amavis_auth_user and $amavis_auth_pass) to unauthenticated forwarded (passed) mail, besides using them for submitting original messages (notifications and quarantine). This is similar to how mailing list managers are allowed to work (rfc2554). Note that the Perl module Net::SMTP in its current version is unable to specify the 'submitter' in its 'MAIL FROM' command, this should be rectified; --------------------------------------------------------------------------- July 1, 2004 amavisd-new-20040701 / amavisd-new-2.0 release notes MAJOR NEW FEATURES (since amavisd-new-20030616-p10): - security improvements: no shell required in chroot jail, checks performed to see if dropping privileges was successful, can drop privileges before reading config file; - the 'amavisd reload' command reimplemented, it now works even if running chroot-ed; - new feature: policy banks hold sets of configuration variables that may be switched with another predefined set based on incoming port number or original SMTP client IP, avoiding the need to run more than one instance of amavisd daemon; - new feature: @score_sender_maps is a soft variant of black- and whitelisting; - extended semantics of SQL field wblist.wb for soft black/white-listing; - redesigned mail structure representation allows better control over 'banned' names and types; - MIME defanging can wrap the entire original mail in a MIME container; - more flexible control on lookups: configurable list of lookup tables observes the specified order of tables and permits arbitrary number of tables of any available type; - level-0 logging either by-message or by-recipient; - syslog priorities are now dynamically derived from the log level; - constantly updates a small database of SNMP-style counters, providing real-time measurements for status monitoring and statistics reporting, reducing the need for analyzing a log file; includes a sample/demo program 'amavisd-agent'; - new 'policy delegation protocol' between helper program and the daemon can pass more information to the daemon and allows the daemon to pass more instructions back to MTA (useful for sendmail milter setup); - persistent cache of recent virus and spam checks, common to all child processes, improves the cache hit rate; - support for IPv6 address formats; - provisional/experimental support for DSPAM spam checker; - support for ClamAV virus scanner via Perl module Mail::ClamAV; - cleaned amavisd.conf : amavisd.conf configuration file with the essentials amavisd.conf-default lists all configuration variables with their defaults amavisd.conf-sample traditional-style commented amavisd.conf with examples ---- INCOMPATIBLE CHANGES since amavisd-new-20030616 (any patch level) - requires Perl module BerkeleyDB with libdb version 3.1 or later (tested with db 4.1); This requirement will be made optional at a later date (in amavisd-new-2.1.0); - a directory at $db_home (default /var/amavis/db) must be manually created to store cache and snmp DB files. It should be writable by user running amavisd. The db files within are removed and re-created at each restart to avoid having to deal with db recovery (but need not be, as far as the program logic is concerned); - sending signal HUP in order to restart amavisd no longer works (previously it only worked in non-chrooted environment and relied on guessing amavisd absolute path); please use 'amavisd reload', or 'amavisd stop' and restart; If the HUP method is really still needed, please replace the line commandline => [], # disable by: commandline => ['/usr/local/sbin/amavisd','-c',$config_file], in file amavisd, adjusting the path if necessary. - due to changed names of temporary files, the old 'DrWebD' av entry needs to be replaced with the current one; - changed $final_virus_destiny default from D_BOUNCE to D_DISCARD - changed $final_spam_destiny default from D_REJECT to D_BOUNCE - changed defaults for variables $virus_quarantine_to, $banned_quarantine_to, $bad_header_quarantine_to and $spam_quarantine_to from undef (no quarantine) to values 'virus-quarantine', 'banned-quarantine', 'bad-header-quarantine' and 'spam-quarantine' respectively. Set them to undef or '' (empty string) to disable quarantine; - add address extension at tag2 level, not at kill level as before; suggested by Jacob Elder and others; - because of the reorganization of lookup tables, a new way of quickly disabling virus or spam checks in amavisd.conf is used. Instead of: # @bypass_virus_checks_acl= qw( . ); # uncomment to DISABLE anti-virus code # @bypass_spam_checks_acl = qw( . ); # uncomment to DISABLE anti-spam code the new recipe is: # @bypass_virus_checks_maps = (1); # uncomment to DISABLE anti-virus code # @bypass_spam_checks_maps = (1); # uncomment to DISABLE anti-spam code - @virus_admin_maps and @spam_admin_maps now take as lookup keys recipient addresses, not sender address as before. The new semantics was often requested, the old semantics was not useful because modern viruses and spam fake sender address, so the choice was made to incompatibly change semantics and use the same config variable names, rather than introduce new names and leave behind useless variables; An implication is that with multiple-recipient mail it is now possible to have more than one admin notification generated when recipients have different admin addresses associated. Still, each distinct admin address receives only one admin notification; - removed old compatibility variable $mailto. Use $virus_admin and $spam_admin variables instead, as suggested in amavisd-new-20021116 release notes, or the more recent inventions @virus_admin_maps and @spam_admin_maps, or their equivalents in policy banks; - removed support for old configuration variable $mailfrom. Use variables $mailfrom_notify_admin, $mailfrom_notify_sender, $mailfrom_notify_recip, and $mailfrom_notify_spamadmin as introduced in amavisd-new-20020630 for the purpose; - removed ancient variable @local_domains, use @local_domains_acl instead, or the more general @local_domains_maps - removed old compatibility code which allowed for a couple of traditional variables to treat value "no" as false. The use of "no" for false has been deprecated since amavisd-new-20021116. - revoke an old compatibility measure where a missing (undef) tag2 level would fall back to the kill level value; - lookup_hash incompatible change (but hardly anyone will notice): a key presence in the Perl hash but with undefined hash value used to be interpreted as true, but now it is treated as undef, causing search to continue with remaining lookup tables (if any). This is more general and more in spirit with other lookup mechanisms; - changed a default to initialize SAVI-Perl every time a child process is started, no longer at master process startup time only. This is to avoid the need to restart amavisd every time the Sophos IDE database is changed. One can revert to the previous behavior by uncommenting a call to Amavis::AV::sophos_savi_init in subroutine fetch_modules_extra; SECURITY - no longer invoke shell to call gzip for compressing quarantined files or to call the sendmail command for submitting messages; the most important consequence is that a shell is no longer needed in a chroot environment and should preferably be removed; - not to forget what was introduced in -p10: inserted a security check for a missing Net::Server patch, and abort if vulnerable; - new command line options '-u user' and '-g group' are available. These are pretty much equivalent to doing a su(1) to the specified user first (in which case the use of these options is redundant). By doing 'su' or by specifying a command-line option '-u username' one can prevent a potential security risk on misconfigured sites where amavisd.conf is writable by UID running amavisd (e.g. not owned by root). If a (non-root) username or UID is specified, privileges are now dropped _before_ opening and evaluating a configuration file. The consequence is that the configuration variables $daemon_user and $daemon_group (in amavisd.conf) can not have an after-effect (a warning is issued if different). If -u is not specified, the behaviour is as before, i.e. the config file is read and evaluated under the current UID (as root unless 'su' was done), and the values of $daemon_user and $daemon_group from the config file are passed to Net::Server, which changes UID during its startup after chroot-ing (if requested). If chroot is desired, the -u must not be used: the root privilege is required to do chroot, and the config file must be read _before_ doing chroot. A case of Catch-22. Be doubly careful of who can modify the configuration file. Another consequence of specifying -u is that any external files (e.g. templates, lookup hashes) as possibly read from amavisd.conf, are now accessed as unprivileged user and no longer as root. The same goes for opening the log file when not logging via syslog. MAJOR NEW FEATURES - policy banks hold sets of configuration variables controlling most of per-message settings, including: static lookup tables, IP interface access rules, forwarding address, log level, templates, administrator addresses, spam trigger levels, quarantine rules, lists of anti-virus scanner entries (or just a subset), banned names rules, defang settings, etc. The whole set of these settings may be replaced with another predefined set based on incoming port number, making it possible for one amavisd daemon to cope with more diverse needs of served user communities which could so far only be implemented by running more than one instance of the amavisd daemon, each with its own configuration file; This mechanism brings new potentials for the future: in principle policy banks could be swapped not only based on port number or SMTP client IP address, but on any characteristics pertaining to a mail message as a whole (not specific to each of its recipients), or to characteristics of a connection from a mailer (e.g. the interface address or protocol); Until a better mechanism is available, a policy bank named 'MYNETS' has special semantics: this policy bank gets loaded whenever MTA supplies a SMTP client IP address (Postfix XFORWARD extension or a new AM.PDP protocol) and that address matches the @mynetworks list. A hash %$interface_policy is a current mechanism of assigning a policy bank to an incoming TCP port number (port must be in the list @$inet_socket_port). Whenever the connection from MTA is received, first a built-in policy bank with an empty name - the $policy_bank{''} gets loaded, which bringings in all the global/legacy settings. Then it is overlaid by whatever configuration settings are in the bank named in the $interface_policy{$port} if any, and finally the bank 'MYNETS' is overlaid if it exists and the SMTP client IP address is known (by XFORWARD SMTP extension command from MTA) and it matches @mynetworks. See amavisd.conf-sample for examples. When a new policy bank is overlaid over an existing set of configuration variables, the variables not present in the new policy bank retain their value. The built-in policy bank (with empty name) is predefined, and includes references to most other variables (the dynamic config variables), which are accessed only indirectly through the currently installed policy bank. Overlaying a policy bank with another policy bank may bring in references to entirely different variables, possibly unnamed. Configuration variables are referenced from a built-in policy bank (which is implemented as a perl hash, i.e. an associative array) by keys of the same name, e.g. { log_level => \$log_level, inet_acl => \@inet_acl, ...}. For scalars one level of indirection is allowed, e.g. a policy bank { log_level => \$log_level }; $log_level=2; is equivalent to { log_level => $log_level } or { log_level => 2 }, but in the first example with an indirect reference, the $log_level may be assigned to even _after_ the policy bank has already been formed. One word of caution: the syntax of entries within a policy bank hash is slightly different from assignments to configuration variables. This is because entries within policy bank are not asssignments, but key=>value pairs as in any Perl hash. And these pairs are delimited by commas, unlike statements, which are delimited by semicolons. Value is separated from its key by '=>' (or by a comma), whereas the assignment operator is '='. Keys of a policy bank are without leading $ or @ or %, unlike variable names. Values of a hash can only be scalars (e.g. strings or refs to arrays). Compare: - value of a policy bank is a reference to a Perl hash, e.g.: { log_level => 3, forward_method => 'smtp:[127.0.0.1]:10025', spam_admin_maps => ["spamalert\@$mydomain"], } - normal assignments look like: $log_level = 3; $forward_method = 'smtp:[127.0.0.1]:10025'; @spam_admin_maps = ("spamalert\@$mydomain"); And a final note: Perl can detect and report typing mistakes in variable names, but mistyped key is just some unused hash entry lurking in a hash, never used and never reported as mistyped/useless. - @score_sender_maps is a soft variant of black- and white-listing, contributing positive or negative score points based on sender e-mail address. Btw, the 'score' in the name '@score_sender_maps' is meant as a verb, recipient is scoring a sender (= to grade, to determine the merit of); Whitelisting is becoming deprecated because it is often and easily abused (but blacklisting can still be useful); # ENVELOPE SENDER SOFT-WHITELISTING / SOFT-BLACKLISTING # Instead of hard black- or whitelisting, a softer approach is to add # score points (penalties) to the SA score for mail from certain senders. # Positive points lean towards blacklisting, negative towards whitelisting. # This is much like adding SA rules or using its white/blacklisting, except # that here only envelope sender addresses are considered (not addresses # in a mail header), and that score points can be assigned per-recipient # (or globally), and the assigned penalties are customarily much lower # than the default SA white/blacklisting score. # # The table structure is similar to $per_recip_blacklist_sender_lookup_tables # i.e. the first level key is recipient, pointing to by-sender lookup tables. # The essential difference is that scores from _all_ matching by-recipient # lookups (not just the first that matches) are summed to give the final # score boost. That means that both the site and domain administrators, # as well as the recipient can have a say on the final score. - the MIME defanging (defang: to make harmless or less powerful) wraps the entire original mail in a MIME container of type 'Content-type: multipart/mixed', where the first part is a text/plain with a short explanation, and the second part is a complete original mail, enclosed in a 'Content-type: message/rfc822' MIME part. Defanging is only done when enabled (selectively by malware type) and the malware is allowed to pass (*_lovers or *_destiny=D_PASS) Conventional mail header fields are retained, and header fields Resent-{From,Date,Message-ID} are added. A header field X-Amavis-Modified is inserted to indicate that the mail body has been modified. It is an experimental feature (disabled by default, except for mail bombs), and not very efficient for large mail. MIME defanging is unconditionally done for mail bombs, i.e. when the X-Amavis-Hold header field is added. The text in the first MIME part describes the reason, e.g.: WARNING, possible mail bomb, NOT CHECKED FOR VIRUSES: Exceeded storage quota 29089500 bytes by do_unzip; ... When MIME defanging is enabled for passed spam, the first MIME part contains the full SpamAssassin report. The MIME defanging feature is not available in the sendmail milter setup. - new parameter: @mynetworks It is an IP access list which determines if the original SMTP client IP address belongs to our internal networks. It is much like the Postfix parameter 'mynetworks' in semantics and similar in syntax, and its value should normally match its Postfix counterpart. It only affects the value of a macro %l (=sender-is-local), and the loading of policy 'MYNETS' (if present). Note that '-o smtp_send_xforward_command=yes' (or its lmtp counterpart) must be enabled in the Postfix service that feeds amavisd, otherwise client IP address is not available to amavisd-new and new features based on @mynetworks do not work (the %l macro works as before and bases its decision on sender e-mail address matching local_domains); LOGGING - syslog priorities are now dynamically derived from the log level of each log message (the first argument of sub do_log). The priority as specified in the $SYSLOG_LEVEL configuration variable is ignored in 2.0 (no longer in 2.1), but the 'facility' is not ignored. This makes it possible to influence the log verbosity by syslog.conf settings. Here is an example of a useful syslogd.conf setting (some levels may be left out; assumes the $log_level is high enough, e.g. 2, to produce any low-priority log messages at all): mail.err /var/log/messages mail.notice /var/log/amavisd.log mail.info /var/log/amavisd-info.log mail.debug /var/log/amavisd-debug.log (On linux do not forget to prefix filenames in syslog.conf with a '-' to disable sync, which has much impact on syslog performance!) At the moment the mapping of message log levels to syslog priorities is hardwired: level <= -3: LOG_CRIT level <= -2: LOG_ERR level <= -1: LOG_WARNING level <= 0: LOG_NOTICE level <= 1: LOG_INFO (in version 2.0) level <= 2: LOG_INFO (since 2.1) else: LOG_DEBUG Not to be confused with the $log_level configuration variable setting, which still works as before, suppressing generation of all log messages with log levels above $log_level. For efficiency reasons one should not specify unnecessarily high $log_level and then discard low syslog priority messages in the syslogd. - added $log_recip_templ variable, which is similar to $log_templ, but gets called for every recipient (the $log_templ is evaluated only once per message). Normally one or the other log template should be disabled by assigning undef or an empty string to the corresponding variable; - the default $log_templ no longer shows quarantine ID if quarantining is disabled; - added a macro %. (a dot), which might be useful in the $log_recip_templ; Its value is empty when $log_templ is expanded, and is a recipient counter (starting by 1) when $log_recip_templ template is expanded. Based on this macro one can provide a single template for both the $log_templ and the $log_recip_templ if needed, or perhaps let the log entry for the first recipient be more verbose that the rest; - added a macro %T which expands to a list of triggered SA tests, but only when $log_templ and $log_recip_templ are expanded. In notifications the %T is still a list of To: addresses. An overlaid semantics, but we are running out of letters and a macro expander rewrite would be needed; - new macros %k, %1, %2, %O, please see README.customize LOOKUPS - new configuration variables make it more flexible to specify arbitrary list of lookup tables. Legacy configuration variables are still available and are referenced from the default values of @*_maps lists. If these lists are redefined, legacy variables are not used. The new variables (lists) are: @local_domains_maps @bypass_virus_checks_maps @bypass_spam_checks_maps @bypass_banned_checks_maps @bypass_header_checks_maps @virus_lovers_maps @spam_lovers_maps @banned_files_lovers_maps @bad_header_lovers_maps @virus_admin_maps @spam_admin_maps @virus_quarantine_to_maps @banned_quarantine_to_maps @bad_header_quarantine_to_maps @spam_quarantine_to_maps @spam_quarantine_bysender_to_maps @banned_filename_maps @spam_tag_level_maps @spam_tag2_level_maps @spam_kill_level_maps @spam_dsn_cutoff_level_maps @spam_modifies_subj_maps @whitelist_sender_maps @blacklist_sender_maps @score_sender_maps @addr_extension_virus_maps @addr_extension_spam_maps @addr_extension_banned_maps @addr_extension_bad_header_maps @keep_decoded_original_maps @map_full_type_to_short_type_maps @viruses_that_fake_sender_maps - for more informative logging of lookup operations, a new object type Amavis::Lookup::Label can be inserted to lists of lookup tables for the purpose of labeling the main purpose of the list; - all lookup* subroutines can now return matching key when called in a list context; - lookups can now return a list of all matching entries (not just the first match); used for the new soft- white/blacklists (@score_sender_maps); - sub lookup() now allows for one level of list elements dereferencing, which makes possible the construction of the argument list and later still be able to modify its members (e.g. creation of regexp lookup table objects in the configuration file); It facilitates transition from old hard-wired lists of lookup tables to new @*_maps list variables which permits specifying an arbitrary number of lookup tables and to specify their search order; - simplify and unify calls to lookup() by collecting arguments (references to lookup tables) in lists, e.g. @local_domains_maps, @virus_lovers_maps, @virus_admin_maps. These array variables default to lists of legacy variables, which are never directly used by the program. Either the individual legacy variables may be assigned to, or the entire list replaced, in which case the legacy variables no longer have any effect. - lookup_acl: respect $localpart_is_case_sensitive setting; - lookup_hash and lookup_sql: rewritten lookup_hash and factored out the common code from lookup_hash and lookup_sql to make_query_keys(); - lookup_hash bug fix: avoid splitting address literal as if it were a domain name; (a bug with key '.' not being tried for address literals fixed thanks to Uwe S. Fuerst); SQL, LDAP LOOKUPS - extended semantics of SQL field wblist.wb, which can hold a score value boost, which is interpreted as soft black/white-listing (the same semantics as the value in @score_sender_maps); - recognize SQL server error 2013/"Lost connection to" and treat it the same as 2006/"MySQL server has gone away"; by Max Kalika; - full domain stripping: @.sub.example.com @.example.com @.com @. - lookup_hash and lookup_sql: limit the list of subdomain search keys to 10 levels as a sanity measure; e.g. for address user@14.13.12.11.10.9.8.7.6.5.4.3.2.com the subdomains keys search list starts at .9.8.7.6.5.4.3.2.com; (domain names are limited by standard to 127 levels); - prepare SELECT statements on demand; - recognize all-zero and all-null boolean fields as false; - recognize new (optional) fields in the table 'policy': spam_dsn_cutoff_level virus_quarantine_to, banned_quarantine_to, bad_header_quarantine_to addr_extension_virus, addr_extension_spam, addr_extension_banned, addr_extension_bad_header (the addition of virus_quarantine_to was suggested by Harald Kapper); - consider the SQL user id a string (no longer required to be numeric); thanks to Max Kalika / Gentoo support; - LDAP white/black list support by Jacques Supcik (similar to contribution from Scott Dier and Eric Dorland, which I forgot about, sorry); - added amavisSpamQuarantineTo to the LDAP schema; new version of LDAP schema; by Jacques Supcik: In the previous schema, the tag levels have been defined as integer. This is too restrictive, and have now been changed to strings (there is no float type in LDAP); - added LDAP attributes for completeness: amavisBadHeaderLover, amavisBypassBannedChecks, amavisBypassHeaderChecks, amavisVirusQuarantineTo, amavisBannedQuarantineTo, amavisBadHeaderQuarantineTo; by Jérôme Schell; DECODING / DEARCHIVING - provide optional ability to retain complete email message in its un-decoded form (alongside its decoded parts) for a virus scanner to see (enabled if $keep_decoded_original_re matched string 'MAIL'); suggested by Tomasz Papszun; (partly backported to amavisd-new-20030616-p8); - rewrite code that generates new file names (Amavis::Unpackers::Part->new), and rewrite code dealing with banned names. Keep information about each part organized as a tree, matching the descendence of each part, new package/object Amavis::Unpackers::Part to collect such information; - make Amavis::Unpackers an optional module: the interface code to external decoder/unpackers/dearchivers does not get compiled and does not consume virtual memory if $bypass_decode_parts is true; (previously it just didn't get called, but was sitting in memory nevertheless); - decode RPM archives by converting them to cpio, if rpm2cpio and cpio are available; - do_tnef: extract $tnef->message if it exists, not just $tnef->attachments; - support extracting MS cabinet files (CAB) by calling cabextract, if enabled and found. Beware: Lars Hecking warns that cabextract 0.6 is quite buggy and the author has been notified. VIRUS SCANNING - support for ClamAV virus scanner via Perl module Mail::ClamAV, based on code by Roberto Pereira da Rosa; - don't call virus scanners if there are no files in the directory to be scanned (e.g. mail with an empty body); some virus scanners don't like to be given an empty directory (e.g. Symantec newer savsecls); reported by Marco Bicca; - rewritten/unified/generalized subroutines ask_daemon and sophos_savi based on the new subroutine ask_av; - scan parts directory for file names exactly once regardless of the number of virus scanners and their arguments (containing '{}/*' or not); - supply full original mail to virus scanners in case of MIME parse errors (in addition to any possibly decoded parts); - when collecting file names to be virus scanned, prepare a hash which maps base file names to Amavis::Unpackers::Part objects, and makes it available to virus scanner interface routines, which may benefit from the additional information about the file to be scanned. In particular, the new interface to Mail::ClamAV now turns on the option CL_MAIL, and the interface to SAVI turns on the option MIME, when entire mail is passed to AV scanner for checking. This enables ClamAV and SAVI to attempt MIME decoding the file by itself. TODO: The same option (--mime) would need to be specified when calling 'clamscan' and supplying a non-decoded mail for checking; pointed out by Riccardo Ghiglianovich and Michael Boelen; OTHER EXTERNALLY VISIBLE CHANGES - The 'amavisd reload' command is now implemented differently: old: signals SIGHUP to a running amavisd process and exits immediately; the running amavisd process (under control of Net::Server) when it receives a SIGHUP starts its own copy with same arguments and exits; new: signals SIGTERM to a running amavisd process, waits for it to finish, then continues (same as 'amavisd start') to become a new daemon; The new method works even when chrooted, and is more reliable when the existing process is slow to terminate, as it actively waits for the previous daemon to finish before proceeding to promote itself to become a new daemon. - a simple demo program 'amavisd-agent' is provided, allowing for continuous inspection of SNMP counters; a path to the /var/amavis/db is hardwired in the program, modify it if necessary. - server-side support for optional Postfix SMTP/LMTP command XFORWARD: information about the original SMTP client IP, its DNS name, HELO name and protocol used is now made available to the amavisd program for logging and other purposes. The same information can also be obtained from the 'Amavis policy delegation protocol (AM.PDP)' if the helper program supports it (useful for sendmail milter setup); - client-side support for optional Postfix SMTP command XFORWARD: if MTA announces in its SMTP EHLO response that it supports XFORWARD, amavisd will provide additional information about the original SMTP client if the information is available (either from XFORWARD on the receiving side, or by the 'Amavis policy delegation protocol'; - server side support for the new amavis helper protocol which allows for header modifications, removal of recipient addresses (e.g. non- spam lovers) or rewrite of recipient addresses (e.g. adding address extensions), and specification of full SMTP response; - modified search logic for matching mail parts against $banned_filename_re; The old search order for names did not result in what one might expect when pattern list elements with a value of false were used in $banned_filename_re. Namely, all three components were searched independently (Content-Type, declared name, and file(1) type) and a logical or was used. Because searches for each mail part were independent, it was not possible to specify for example that anything within a zip would be allowed. If any of these searches returned true, mail was blocked. To make this useful, a complete rewrite of mail unpacking was needed and all information be made available in one place after the unpacking is over, so that checking for banned names can be done all at once. The search order is now much the same as used in rsync and its server, see man rsync, section 'EXCLUDE PATTERNS'. The new comments in amavisd.conf-sample explain the new situation. - replaced $relayhost_is_client by a more flexible specification: To make it possible for several hosts to share one content checking daemon, the IP address and/or the port number in $forward_method and $notify_method may be specified as an asterisk. An asterisk in the colon-separated second field (host) will be replaced by the SMTP client peer address (i.e. the MTA host). An asterisk in the third field (tcp port) will be replaced by the incoming SMTP/LMTP session port number plus one. This obsoletes the previously used less flexible configuration parameter $relayhost_is_client. An example: $forward_method = 'smtp:*:*'; $notify_method = 'smtp:[127.0.0.1]:10025'; The same functionality can also by achieved by using a bigger hammer, the policy banks. These may completely replace the global settings for $forward_method and $notify_method, based on incoming port number; - turn address extension variables (the so called "plus addressing") into recipient-based lookup tables, including the SQL lookups. For example: @addr_extension_virus_maps = ('virus'); # defaults to empty @addr_extension_spam_maps = ('spam'); # defaults to empty @addr_extension_banned_maps = ('banned'); # defaults to empty @addr_extension_bad_header_maps = ('badh'); # defaults to empty or perhaps: @addr_extension_virus_maps = ( {'sub.example.com'=>'infected', '.example.com'=>'malware'}, 'virus' ); suggested the Gentoo modification, Jacques Supcik, and others; - log and report hits and tag/tag2/kill levels rounded to three decimal places (trailing zeroes trimmed), no longer rounded to one decimal place; - added @spam_dsn_cutoff_level_maps, making it possible to specify different DSN cutoff levels for different recipient domains or users. In multi-recipient messages where recipients can specify different values, the maximum value is used for deciding whether DSN should be suppressed; suggested by Ales Casar; - configuration variable $gets_addr_in_quoted_form is no longer used; knowledge about address form (quoted or not) is now implicit in the receiving protocol; - if tag level turns out to be undef, it will not be shown in X-Spam-* header fields, and will be interpreted as having a value lower than any spam score when deciding whether to insert X-Spam-* header fields or not; - added macros %a and %g: * %a original SMTP session client IP address (empty if unknown) * %g original SMTP session client DNS name (empty if unknown) (like macros %I and %M that were once proposed by Dibo in his 2002-07 patch) This information may be available from Postfix when XFORWARD protocol extension to SMTP is enabled, and can be made available by helper program (e.g from sendmail milter) when the new AM.CL protocol is used; - added macro %p, expanding to a current policy bank name (or empty if a built-in policy bank is still in place); - added macro %r, expanding to the contents of the first Resent-Message-ID header field, or empty if no such field exists. Include reporting the Resent-Message-ID in the log and in the sender notification; suggested by Oliver Gorwits; - new configuration variable $addr_extension_bad_header for completeness; - added $bad_header_quarantine_to, @bad_header_quarantine_to_maps, $bad_header_quarantine_method, and $warnbadhrecip for completeness; suggested by Robin Lynn Frank; - MIME::Parser errors now contribute to bad-header checks, so that the header checking is now conceptually extended to MIME sub headers (Postfix similarly considers MIME subheaders part of mail headers); MIME::Parser 6.1xx or later is recommended. - allow $*_quarantine_method to be undef as a quick way of disabling some quarantine (it also can be disabled as previously, by using method 'local:' and following its rules); - persistent cache of recent virus and spam checks, global to all child processes, can improve the cache hit rate. Uses BerkeleyDB database (hash and queue) and its interlocking mechanisms (Berkeley DB Concurrent Data Store) for the purpose. The V3.1 or better is required, V4 is recommended. - include version information in the 'Usage' text; - rewritten lookup_ip_acl() and added ip_to_vec() to allow for IPv6 address syntax as specified in rfc3513 to be used in IP lookups; - @inet_acl now defaults to ('127.0.0.1', '::1'), i.e. it adds the IPv6 loopback address to the list; - new configuration variable $sa_spam_level_char (defaults to '*') allows specifying another character for X-Spam-Level bar. Empty or undefined value disables inserting the X-Spam-Level header field; - added configuration variable $sa_spam_report_header to enable/disable inserting the X-Spam-Report header; patch by Craig Sanders; - added $banned_quarantine_to configuration setting to make possible the quarantining of banned mail to a different place from viruses; - don't insert virus-, banned- and bad headers- related headers for passed mail to recipients with corresponding bypass_*_checks, making them believe the mail was not spam-checked (as they are not expecting such headers anyway); This was already done in version amavisd-new-20030616-p6 but only for spam-related headers; - for choosing address extensions use the same criteria as for adding header fields, e.g. pretend to not know the result of a certain test (virus, spam, ...) when recipient chooses to bypass such test, even if the result of the test is known; - added variable $sa_spam_subject_tag1 (undef by default). If $sa_spam_subject_tag will not be inserted (at tag2 level), and $sa_spam_subject_tag1 is nonempty, this string (e.g. '***possible SPAM*** ') will be inserted into the Subject header field for spam levels above tag level; suggested by Immo Goltz; - added separate configuration variables $banned_files_quarantine_method and $bad_header_quarantine_method. Quarantining of banned files and bad headers were previously controlled by $virus_quarantine_method; - rewritten read_hash, it is now possible to specify key value (right-hand side) for each key. If value is not specified, a '1' is assumed as before; - SMTP server support for rfc2554 authentications (PLAIN and LOGIN only) allows client authentication to be relayed to the MTA when message is forwarded. Might be useful if amavisd-new is used in a Postfix SMTP proxy setup, but is not needed for other setups. Disabled by default, see variables $auth_required_inp and @auth_mech_avail. - SMTP client support for rfc2554 authentications (any authentication method as supported by Net::SMTP and Authen::SASL Perl modules). Authentication of forwarded mail (PLAIN and LOGIN only) is carried over from the incoming mail, authentication to be used when submitting notifications is controlled by configuration variables $amavis_auth_user and $amavis_auth_pass. Disabled by default, see variable $auth_supported_out. - provisional/experimental support for DSPAM spam checker (pre 3.0): if configuration variable $dspam is nonempty and represents a path to a 'dspam' program, a message is passed to dspam and its inserted headers of the form X-DSPAM-* are axtracted and then made available for SpamAssassin rules to check and score if desired. All messages are currently presented to dspam as the same user, affecting how its database is being built. False negatives and false positives (based on SA assessment) are fed back into DSPAM as a simple form of auto-learning. Works reasonably well, but do not expect miracles. See subroutine spam_scan. Here is how DSPAM can be installed to be able to be used by amavisd-new: dspam 2.x: ./configure --enable-alternative-bayesian \ --with-userdir=/var/amavis/dspam --enable-signature-headers \ --without-local-delivery-agent --without-quarantine-agent dspam 3.0.0: ./configure \ --with-dspam-home=/var/amavis/dspam --enable-signature-headers \ --without-delivery-agent --without-quarantine-agent make install chmod u-s,a+rx /usr/local/bin/dspam chown vscan:vscan /var/amavis/dspam and the following can be inserted into the SA config file (local.cf) to make it recognize and incorporate DSPAM's assessment: header DSPAM_SPAM X-DSPAM-Result =~ /^Spam$/ describe DSPAM_SPAM DSPAM claims it is spam score DSPAM_SPAM 0.5 header DSPAM_HAM X-DSPAM-Result =~ /^Innocent$/ describe DSPAM_HAM DSPAM claims it is ham score DSPAM_HAM -0.1 Eventually DSPAM support should be removed from amavisd-new, as soon as SA will be able to call it on its own. INTERNAL CHANGES - reformatted the whole program, reducing indentation from 4 to 2 and replacing TABs with spaces (with some dubious help from perltidy, plus lots of manual adjustments); - completely rewritten code to handle both the old and the new amavis helper protocol, as well as Postfix 'TCP client/server table lookup protocol' as specified in the Postfix documentation: tcp_table(5); (process_policy_request, prepare_policy_query, check_amcl_policy) - type_short may now be a list of short types. Typical use is to classify a MS executable as both an 'exe' and as 'exe-ms', which makes more specific banned rules possible without unnecessary complication in regexps; - parts now carry attributes, which can be inspected for banned checks; current attributes are U for undecodable, and C for (en)crypted; - opened another can of Perl worms (taint bugs): turn on Perl pragma "use re 'taint'" in all modules, and selectively turn it off where needed. It replaces cumbersome manual preservation of taintedness when regexp saved ranges are used without intention to untaint. Because of Perl bugs, strategically placed local($1,$2,...) are needed, otherwise previous taint flag in $1, $2, ... can be brought on to new variables, which can all of a sudden become tainted out of nowhere; - catch and report throws (die) in pre_loop_hook() to properly report problems during initialization; - introduced subroutine exit_status_str and unify reporting of subprocess status; - enhanced sub best_try_originator to ignore IP addresses from private, local and dedicated IANA networks (rfc3330) and look for the first public address in the 'Received' path; - examine first four (chronologically) Received header fields (instead of first two) when looking for an originator IP address, and ignore those with private IP addresses; - moved code dealing with body digest cache to a new package Amavis::Cache to facilitate transition to shared or persistent cache; - new explicit cache expiration time controls (time to live in seconds): $virus_check_negative_ttl $virus_check_positive_ttl $spam_check_negative_ttl $spam_check_positive_ttl $cache_entry_ttl - discard cache db ($db_home) and recreate it during restart; - more informative changes of child process $0, which may show in the ps(1) output; - store tempdir of a current message to the Amavis::In::Message object; - gather some statistics about idle time; - reorder and adjust mapping from file(1) results to file type classes; - optimization: instead of invoking file(1) utility program for each part to be analyzed, now call it once for each round of currently available parts, giving it the list of all available parts as arguments; - shorten the names of parts from part-..... to p..., to be able to stash more files names into a command line, e.g. when calling file(1) or external virus checker which can not deal with directories; - use regexp lookup table mechanism (table $map_full_type_to_short_type) to match long types (output of file(1) utility) to short types (.exe, .jpg, .doc, ...). The default table can be replaced by a customized table in amavisd.conf; - replace $(?!\n) with \z in regular expressions throughout; replace most of the remaining $ with \z in regular expressions. The regular expression primitive \z is available since perl 5.005. - TODO: disregard $MAXFILES during initial MIME unpacking; reported by Stephane Lentz and Robert LeBlanc; Some un-edited notes on the new banned rules mechanism: (I wrapped your log lines, and replaced \\ by \ for clarity: | Feb 24 19:07:29 hauptpostamt amavis[29847]: (29847-04-5) p.path BANNED: | | "P=p002,M=application/octet-stream,T=zip,N=document.zip | | P=p003,T=exe,T=exe-ms,N=document.htm .scr", part p003 is of type (file(1)) MS executable, with suggested name "document.htm .scr" (lots of spaces in the name) its parent resides on temp file p002 (i.e. p003 was extracted from it), which is of type (T) zip archive, with suggested (MIME) name (N) "document.zip", and has a MIME type (M) "application/octet-stream". such a component p003 lying within such p002 is considered banned by the following regexp rule (one rule within the $banned_namepath_re list): | matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^./\t\n]* \. | (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)" which says that any component at any level must not have a name (N) matching a pattern: any number of characters, a dot, any number of non-dot and non-slash characters a dot, and ending with: exe or vbs ... (basically: double extension ending with listed extensions) The complications such as using [^\t\n]* instead of .* are there to keep regexp contained within fields and ancestors/descendents. There is one detail to remember when comparing logged p.path log entries and the actual matching rules: - for the sake of readability the logged entry has \n (newlines) converted into ' | '. The \n is a separator between components in the tree from the root (the mail itself, hidden) to the the leaf component which can not be further expanded (i.e. not an archive) - for the sake of readability the logged entry has \t (a tab) converted into comma, separating information fields such as P=... M=.. T=.. M=.. So the above logged string: P=p002,M=appl...,T=zip,N=document.zip | P=p003,T=exe,T=exe-ms,N=document.htm.scr is actually a single string: P=p002\tM=appl...\tT=zip\tN=document.zip\nP=p003\tT=exe\tT=exe-ms\tN=document.htm.scr and a Perl regexp is applied directly to it. The raw string is rather unsightly, but the \n and \t were chosen to minimize clash with valid characters within file names. If a \n or \t is present in a name of the components, such character is converted into a space to avoid clashing with separators. | Feb 24 19:11:58 hauptpostamt amavis[31505]: (31505-01-5) p.path BANNED: | "P=p002,M=application/octet-stream,T=zip,N=paypal.zip | | P=p003,T=exe,T=exe-ms,N=paypal.scr", a MS executable named "paypal.scr" within a zip archive "paypal.zip" | matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. | (exe|vbs|pif|scr|bat|com) (\t.*)? $)" block component at any level with a name (N) terminating by dot followed by any of the listed extensions. | Feb 24 19:18:25 hauptpostamt amavis[32159]: (32159-01-2) p.path BANNED: | "P=p002,M=application/octet-stream,T=zip,N=text.zip | | P=p003,T=exe,T=exe-ms,N=text.txt .exe", a MS executable named "text.txt .exe" (with lots of spaces in the name) within a zip archive named "text.zip" | matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^.\t\n]* \.\n | (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)" blocked by the double-extension rule. | Feb 24 19:30:15 hauptpostamt amavis[1690]: (01690-02-8) p.path BANNED: | "P=p002,M=application/octet-stream,T=zip,N=jokes.zip | | P=p003,T=exe,T=exe-ms,N=jokes.doc .exe", | matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^.\t\n]* \.\n | (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)" same thing --------------------------- --------------------------------------------------------------------------- June 29, 2004 Patch: amavisd-new-20030616-p10 - insert a security check to test for a missing Net::Server patch, and abort if vulnerable; - provide and use our own subroutine q_encode to do the Q-encoding when editing an existing invalid header field with non-encoded 8-bit characters, e.g. when inserting ***SPAM*** or ***UNCHECKED*** into Subject. The MIME::Words::encode_mimeword() does not encode spaces and does not limit encoded words to 75 characters, which violates the RFC 2047 and breaks mail readers; reported by Sebastian Hagedorn and Gregor Hoffleit; - fixed a bug in inserting the tag_level header field, which was missing if sender was blacklisted and tag_level was greater or equal to 0; thanks to Joerg Thaler; - amavis-milter.c * log envelope sender address at the same log level (DBG_INFO) as recipient addresses; * remove the log message "(mlfi_eom) header already present", it was inappropriate, the call to smfi_chgheader succeeds even if no such header was already present; * relax permissions on created directory and temporary file to allow group read access (needed if virus scanner runs under a different user id but within the same group); by Adam C. Migus; * add queue id (when available) to most log messages; - do not preserve evidence just because a message gets an X-Amavis-Hold header field; - when logging directly to a file and started as root, change UID of a log file to $daemon_user to avoid restart problems; based on a patch by Carsten Hoeger and Gregor Weiss; - added configuration variable $first_infected_stops_scan to stop anti-virus scanning when the first scanner detects a virus; the default is false, all scanners in a group are called (as usual); - in the AV entry for clamscan added the option '--tempdir=$TEMPBASE'; - in the AV entry for 'Norman Virus Control v5 / Linux' changed the command name 'nvccmd' into 'nvcc'; correction by Michael Ramke of the Norman Data Defense Systems GmbH; - insert debug reports into sub ip_addr_to_name to help recognize DNS problems; - revoke deleting an existing 'X-Scanned-By' header field, which was introduced in -p8; removing it gets in a way when more than one content filter is being chained; - load a missing SpamAssassin v3.0 module, needed when running in chroot; - contributed (for now in the form of an optional patch): support for the incoming qmqpqq protocol over a TCP socket - to be used with qmail. Apply the provided patch 'amavisd-new-qmqpqq.patch', it updates files amavisd.conf and amavisd in the current directory; by Martin Solciansky; - documentation updates; --------------------------------------------------------------------------- April 2, 2004 Patch: amavisd-new-20030616-p9 - avoid choking on undefined $banned_filename_re; thanks to Ales Casar, Sebastian Hagedorn and E. Falk; - if Subject mail header field got $undecipherable_subject_tag inserted, it would also receive the spam tag $sa_spam_subject_tag unconditionally; fixed, thanks to Francis Stevens; - updates to the @av_scanners list in amavisd.conf: * DrWebD now works with a new DrWeb daemon 4.31, thanks to information provided by Krzysztof Cegielski, DrWeb Polska; * updated BitDefender bdc, thanks to Alfredo Milani Comparetti; * clamscan: use documented option --no-summary instead of the undocumented/old(?) option --disable-summary; by Georgy Salnikov; * updated Kaspersky aveclient and F-Secure fsav, thanks to Tomi Hakala; * recognize that KasperskyLab antivirus in demo mode turns on the bit 0x10 in status; avp and avpdc now use the same two sets of status codes; avp: statuses 3 and 6 moved to infected (to match avpdc); statuses 2 and 5 considered infected; all suggested by Georgy Salnikov; - when original undecoded mail is to be kept for virus scanners (requested by patterns /^MAIL$/ or /^MAIL-UNDECIPHERABLE$/ in $banned_filename_re), the preserved file is now named parts/part-00000 instead of parts/email.txt to preserve the size of the name, upon which some virus scanners depend (e.g. DrWebD); A note to Courier users: due to the way a file name in Courier is created and passed to amavisd-new, it is currently not possible to use the triggers /^MAIL$/ and /^MAIL-UNDECIPHERABLE$/ in $keep_decoded_original_re, or a failure to create a hard link occurs; Thanks to Bowie Bailey for helping to troubleshoot the problem; (this limitation is lifted in amavisd-new-2.2.0) - remove option -c when calling gzip, bzip2, compress, lzop and unfreeze (as has long been done in the development version); the option -c is not needed when no file argument is present, and some implementations of gzip and compress may choke on it; - make loading of Perl module Carp::Heavy optional, versions of Carp that came with Perl 5.005 did not have it; reported by Jefferson Pizzolatti; - load module Mail::SpamAssassin::BayesStore::DBM, required by SA 3.0 running in chroot jail; - look for program 'gcpio' ( $cpio = ['gcpio','cpio'] ), as on OpenBSD the plain cpio does not support required options (the --no-absolute-filenames is essential), but GNU cpio does; thanks to Manfred Gloiber; - relax parsing of file(1) output to allow tab as well as space to follow a file name; the file(1) on Solaris uses tab instead of space; suggested by Glen Harris; - make LHA understand self-extracting archives (SFX); patch by Georgy Salnikov: Although lha unpacks any non-SFX archive independently if its extension, it unpacks the SFX lha archives only if they have the extension .exe. Now the file is symlinked to $part.exe for checking by lha: this will now work for SFX, and will still work for non-SFX. Also, do_lha will return 0 if $exec and the part cannot be de-archived, the same as in do_unrar. The executable file formats are checked for being zip/rar/lha SFX archives. However, if the SFX archive is not a zip archive, do_unrar always returns success, so that if the archive is also not a rar, it will be never checked by lha. Now do_unrar returns 0 if $exec and the corresponding part can not be un-RARed; - do_executable and do_unarj: added checking for SFX arj; by Georgy Salnikov: (commented out the call to do_unarj in do_executable until more experience is gained on how well unarj survives certain mail contents; Mark); - do_unarj: let arj/unarj work on file named part*.arj; - when calling IO::File::open() use '+>' instead of 'w+' to avoid Perl taint bug ($mode turns tainted) (bug still present in 5.8.2) triggered by expression in IO::Handle::_open_mode_string(); - attack the Perl 5.8.0/5.8.1/5.8.3 taint bug (once variables $1,$2,etc get tainted they start spreading taintedness to other variables): * insert local($1,$2,$3,...) in blocks of code which call external modules which trigger the bug (Mail::SpamAssasin, MIME::Parser, ...) * insert local($1,$2,$3,...) in blocks of code which depend on these variables to be clean, and which demonstrated through bug reports and experience with various version of Perl that these variables were not always taint-clean; The last taint incident triggered by SA 3.0.0 (svn) reported by Luc de Louw; - recognize status EX_NOUSER when forwarding via pipe to sendmail (old setup); previously it was treated as a temporary failure; patch by The Mindflayer; - turn error message 'error reading from client socket' back into 'client broke the connection without a QUIT' for consistency with P7; - when %virus_admin lookup table is used, prefer $msginfo->sender_source (unmangled sender domain) over $msginfo->sender; suggested by Pawel Golaszewski; - nicety: in virus recipient notifications now supply the To: header field with the true recipient address instead of "undisclosed-recipients:;" in case of single-recipient mail; suggested by several people; - amavisd would insert differently capitalized header fields (either X-AMaViS-Alert or X-Amavis-Alert), depending on the reason being reported; now use the X-Amavis-Alert throughout; reported by Carsten Hoeger; - fetch Perl auto-loaded modules auto::POSIX::setgid and auto::POSIX::setuid if they exist; they are needed by older versions of Perl when running in chroot. A manual change was needed until now (documented in README.chroot), which should no longer be needed; a FreeBSD problem report 64636. - helper program amavis-milter.c (used in the sendmail milter setup): * amavis-milter.c only included the client IP and client host name on the first mail transaction of a multiple-transaction SMTP session, but not in subsequent transactions; pointed out by Stephane Lentz. The solution was once already provided back in July 2002 by Radoslav Dibarbora - Dibo, and forgotten; credits where credits are due: http://marc.theaimsgroup.com/?l=amavis-tech&m=102569946910555 * make a clear distinction between message data and connection data, which required code reshuffling and revealed previous unlean solutions; * add error checking and reporting to mkdir/rmdir/open/unlink/write system calls; previously an error could pass by unnoticed or just caused a tempfail without an explanation; * change final milter status ACCEPT into CONTINUE to allow further milters in the milter chain to examine the mail; * code cleanup; * regenerated helper-progs/configure with Autoconf 2.57 to make it capable of detecting mkdtemp and still be able to find sendmail libmilter files; thanks to Sebastian Hagedorn for a problem report, for testing, and for revealing a bug in the pre-released version; * use different syslog priority for different internal message log levels; by Sebastian Hagedorn; * adjust log levels of messages, set default verbosity 1 (DBG_WARN); - documentation updates, including the updated comments in amavisd.conf regarding whitelisted senders, to reflect the change indicated in the amavisd-new-20030616-p4 release notes; --------------------------------------------------------------------------- March 9, 2004 Patch: amavisd-new-20030616-p8 - be compatible with SpamAssassin version 2.70 and 3.0 as well as 2.6x; SA changed its API, replacing Mail::SpamAssassin::NoMailAudit with $spamassassin_obj->parse and belonging objects; - as a stop-gap solution to the W32/Bagle-{F,...} detection problem (password-protected zip archives), three new measures are available: * ability to present the full non-decoded original message to virus scanners was partly back-ported from the development version. Enabled by adding qr'^MAIL$' or qr'^MAIL-UNDECIPHERABLE$' to the list in $keep_decoded_original_re, as illustrated in amavisd.conf; similar to a patch by Ted Cabeen. The following keys are used for a lookup into $keep_decoded_original_re: 'MAIL' always provide a full original message to virus scanners (besides its successfully decoded components); 'MAIL-UNDECIPHERABLE' same as for 'MAIL', but only if it contains undecipherable components such as password protected archive members, unsupported compression methods or encrypted parts (e.g. with PGP). Don't put too much trust into this, as some more exotic file formats may not be understood and not flagged as undecipherable; * a key 'UNDECIPHERABLE' is matched against $banned_filename_re when mail contains any undecipherable components, and if lookup returns true, mail will be banned. For example: $banned_filename_re = new_RE( qr'^UNDECIPHERABLE$', # contains any undecipherable components ... ); * a string can be prepended to Subject (for local recipients only) if mail could not be decoded or checked entirely, e.g. due to password-protected archives or non-decodable mail bombs: $undecipherable_subject_tag = '***UNCHECKED*** '; # undef disables it NOTE: this solution is a quick-fix response to popular demand. Although the same or similar functionality will probably remain in future versions, the syntax and exact semantics may be refined. - bring do_unarj in line with the rest of de-archivers: provide the same degree of mail bomb protection that was available in do_unrar; retain original archive for inspection by a virus scanner if it contains any members that can not be extracted (e.g. password protected members or unsupported compression schemes). (The .arj size checking deficiency was mentioned in the AMaViS Security Announcement (ASA) 2004-01-19); - don't call a virus scanner if there are no files (no mail parts) to scan; it caused problems with certain scanners like aveclient, which expect a list of file names as arguments and complain if the list is empty; reported by Daniel Luttermann; - a much needed feature: can specify $sa_dsn_cutoff_level in amavisd.conf to suppress sending a DSN (delivery status notification) when spam level is above this value, effectively turning D_BOUNCE into D_DISCARD for this message; undef disables the feature and is a default. A good first approximation value is 10, or with some risk go down to 8. The parameter has no effect if DSNs are already disabled (e.g. when $final_spam_destiny is D_DISCARD or D_REJECT); - do_unrar: double check the archive size (against summary line as well); - derive the value of macro %l from sender_source instead of from a declared sender address, so as to not rise false alarms when sender address is known to be faked; patch by Joerg Friedrich; - reduce the number of retries to connect to a daemonized virus scanner from 3 to 2, so that a fallback to backup scanners occurs sooner; - updated $banned_filename_re example in amavisd.conf, modify it to will; - updates to the @av_scanners list in amavisd.conf: * KasperskyLab AVP - aveclient: *IMPORTANT* updated entry for the exit status only reflects the result of the last file scanned, we must use regular expression to detect viruses; fix provided by Andreas Triller; * Trend Micro FileScanner - vscan: added option -za, otherwise some broken archive may sneak-in a virus; suggested by Stephane Lentz; * Sophos sweep: added options -cab -tnef --no-reset-atime; * Dr.Web command line scanner: updated options, recognize exit status when using evaluation license; * ClamAV clamd: change socket location to a more usual value, adjust to will; - amavisd.conf: added new virus names to the $viruses_that_fake_sender_re list, and uncommented the [qr/.*/=>1] line, so that by default any unknown virus will be treated as a sender-faking virus; adjust to will; - do not send bad header notifications in response to messages from mailing lists; - added header check for folded header field lines made up entirely of whitespace (a 'header space gap' violation to rfc2822); a check is only made when other header checks (bad character in header) are enabled; - distinguish an empty string from undef in $mailfrom_to_quarantine, making it possible to specify a null return path when quarantining to a mailbox; - helper-progs/amavis-milter.c (sendmail) enhancement based on work by Stephane Lentz: * the name of a temporary directory is derived from the sendmail queue ID, making it easier to match sendmail and amavisd-new log entries; the queue ID is also a part of the quarantine file name; * a phantom 'Received:' header field is prepended on the temporary file to preserve the information on the original SMTP client IP address, host name and queue ID. This trace header field does not propagate to recipients (is not inserted into the original message), but is available in the quarantined messages and is visible to SpamAssassin. Adding a milter macro {b} on ENVFROM is advised to preserve the MTA timestamp in the log, although not mandatory (falls back to current time): define(`confMILTER_MACROS_ENVFROM', confMILTER_MACROS_ENVFROM``, {b}'')dnl If the new amavis-milter.c gives you trouble, switch the soft link to the previous version in the same directory. Keeping macro {b} does not hurt. - catch possible errors in pre_loop_hook and report them properly; - better check for I/O errors on SMTP input socket; - add the following modules to the list of pre-fetched modules: Carp::Heavy, IO::Handle, IO::Socket::UNIX, IO::Socket::INET. The absence of Carp::Heavy could mute error report or a backtrace when running chroot-ed; the rest are for completeness only; - added a macro %z which expands to the original mail size (in bytes); it could be useful in the $log_templ; thanks to Nick Leverton; - make SpamAssassin timeout value configurable: variable $sa_timeout; by Henrique M. Holschuh (Debian); - 'Received:' header field cosmetics: use 'unknown' in case the HELO argument was empty, and put-in a 'unix socket' if message was received from a helper program; by Henrique M. Holschuh (Debian); - remove more pre-existing X-Spam* header fields from other scanners: X-Spam-Tests, X-Scanned-By; by Henrique M. Holschuh (Debian); - helper-progs/amavis.c updates (old sendmail setup): report errors, log to syslog, change default dir to something obvious for bug reporting; by Henrique M. Holschuh (Debian); - updated documentation; --------------------------------------------------------------------------- January 5, 2004 Patch: amavisd-new-20030616-p7 - do_unzip and do_unrar: retain an archive if any of its components is password protected or encrypted (plus unpack what can be unpacked, as before). This gives virus checkers a chance to examine the original unpacked archive as a whole (e.g. scanning for variants of W32/Mimail), matching it in non-decoded form against virus patterns even if containing password-protected components. As a consequence, some virus scanners may now log their complaint when encountering such protected archives which previously didn't reach them. Such log entries should be considered informational only; - add module Net::DNS::RR::AAAA to the list of Perl modules to be fetched before chroot takes place; thanks to Per olof Ljungmark; - preload Perl modules DBD::*, based on @lookup_sql_dsn; required when using SQL lookups from a chroot jail; - updated example dsn in @lookup_sql_dsn (file amavisd.conf) to use the new DBD::mysql syntax, and to show how to force accessing SQL server via inet socket which makes it easier to use from a chroot jail; - disregard cached spam results for mail with small body. The most pronounced undesirable effect was in mail with an empty body where spam score was derived from header only; - change the default value for local_domains_sql lookup for the catchall key '@.' under conditions: when using SQL lookups and user record with key '@.' is present in the database and a field 'local' is not present. Previously it surprisingly defaulted to true, now it falls back to static lookup table defaults, the same as if the record '@.' were not present in the table; - fixed ugly text formatting in recipient notifications template; reported by Florian Effenberger; - fixed parsing of 'Received:' header field for some unusual cases; - updated amavisd-new-courier.patch to apply cleanly against -P6 (now P7); by Martin Orr; - added av_scanners entry for 'AVG Anti-Virus', kindly provided by Grisoft s.r.o. from Czech Republic, http://www.grisoft.com/ - make Dr.Web Daemon av entry work with evaluation or regular license; thanks to Andrew I Baznikin; - added av_scanners entry for new AVP client (aveclient) that is shipped whith avp 5.x.x.x. Moved kavscanner to the @av_scanners_backup list (it is presumably slow and less reliable than aveclient), updated other AVP/Kaspersky entries; thanks to Nabil Sefrioui; - added status codes 10 and 15 to the list of ok statuses for kavscanner (10=Password-protected archives, 15=Corrupted files); - add an example dummy virus scanner 'all-clean' to the @av_scanners_backup list which always succeeds, always returning false (= status clean). Uncomment it if desired to avoid mail requeue when all other scanners fail, and to just pass the mail unchecked; - disregard rfc2821 recommendation that 552 smtp response code should be treated as 452. It is unnecessary in amavisd-new setups, and it is wrong because 552 has other meanings assigned to it besides "too many recipients"; - allow IP address in $forward_method and $notify_method to be bracketed, which is needed for IPv6 addresses containing colons. Both formats are now allowed: 'smtp:127.0.0.1:10025' and 'smtp:[127.0.0.1]:10025', the later is now preferred; - change IP address bracketing in log entries 'SEND via SMTP' and 'FWD via SMTP' from [127.0.0.1:10025] into [127.0.0.1]:10025 to match the syntax in the connect log entry and to facilitate parsing of IPv6 addresses from the log; - update received_line() to generate valid Received header field even for IPv6 client addresses; - modified helper program amavis.c to allow it to be run non-root and to set temp file mode for group accessibility. Change its default log level to DBG_WARN; - fix an incorrect SELECT example in README.lookups; thanks to Nabil Sefrioui; - documentation updates; --------------------------------------------------------------------------- November 10, 2003 Patch: amavisd-new-20030616-p6 - change SQL lookup code to better handle SQL database server restarts. After a SQL server restart amavisd-new would previously TEMPFAIL (4xx) all messages until amavisd child process would run down, then resume normal operation with the new child birth. Now the SQL server reconnect is done when the next mail arrives, so only one mail with each amavisd child process TEMPFAILs during SQL server restart, and normal operation resumes faster; - in flatten_and_tidy_dir and in do_ascii: sanitize protection of files and directories which may otherwise be made inaccessible to virus scanners; based on patch by Henrique de Moraes Holschuh (Debian amavisd-new support); problem reported by Aspa and by Tomasz Papszun; - fix a potential security problem: don't let rmdir_recursively and rmdir_flat follow symbolic links; this might be exploited by attempting to delete some foreign file using privileges of the amavisd process (which should not be root); - use cpio option -d when unpacking cpio archives; - don't insert spam-related headers for passed mail to recipients with bypass_spam_checks, making them believe the mail was not spam-checked (as they are not expecting such headers anyway); - text added to the banned mail sender notification template (at the end of file amavisd), explaining to the sender what happened and how to avoid the restriction; edit to will; - amavis-milter.c program in the helper-progs subdirectory is now based on the most recent version 1.1.2.3.2.40 from the AMaViS CVS (maintained by Lars Hecking), but hacked a bit to make its options mostly compatible with the previous version. Start it with option -h to get current usage text. See helper-progs/README if using sendmail milter setup; Please revert to the old one if the .40 gives you trouble; both version are included in the helper-progs subdirectory. - fix parsing of unrar info lines; - consistency with other virus/banned logic: don't send recipient notification (reporting banned name) if mail contains both a virus and a banned name, but $warnvirusrecip is false; reported by Nathan G. Grennan and Urska Brinar; - check for possible I/O errors when reading from SMTP socket, and distinguish error condition from normal TCP session teardown; - do not redirect stderr to /dev/null when calling file(1). This way the diagnostics from file(1) will now at least be visible in the debug session; - determine_file_types: make 'bzip compressed' pattern match older bzip format (v1) as well; thanks to Davaeron; - sub run_command: explicitly close STDERR before reopening it; this way the reporting of possible problems in each operation would be separate, and it seems to avoid a rare problem (open STDERR '/dev/null' failing) reported by Sam Hart; - don't attempt to do lookups in regexp table $viruses_that_fake_sender_re if it is undefined; reported by Chris Paul; - remove existing X-Spam-Score along with other X-Spam* header fields if spam scanning is enabled; - do not skip inserting 'X-Spam-Status: No, hits=- ... WHITELISTED' for whitelisted senders which caused SA to be bypassed - artificially assume score is -10 for the purpose of comparing it to tag_level; reported by Mike Vanecek; - explicitly qualify wblist.rid field in $sql_select_white_black_list just in case someone has a field 'rid' in the other table; - fix showing the value of LC_TYPE environment variable in the log, show LC_CTYPE as well; - SAVI-perl: remove MIME option from the default set of options. Even with recent versions of Sophos SAVI there are cases where the library goes into a spin while trying to decode broken MIME message (same applies to Sophie - one may want to change its configuration); - updated vfind entry in @av_scanners to work with the new version, changing '--vexit {}' to '--vexit {}/*'; thanks to Lowell Filak; - updated kavdaemon entry in amavisd.conf; thanks to Michael Hall and Daniel Melanchthon; - added option '-packed' to 'FRISK F-Prot Daemon' entry; thanks to Manfred Gloiber; - NAI uvscan entry: commented pre/post actions in the entry to show how to remove environment variable LD_PRELOAD after finishing, suggested by David Tilley; added option --mime, suggested by Max and Kevin W. Gagel; added note on how to treat password-protected files as viruses, by Seth Parker; - added 'Dr.Web Daemon' entry which talks directly to Dr.Web daemon over its Unix socket, speeding up a single-file check more than 200 times; provided by Andrew I Baznikin; - another entry for Symantec AntiVirus Scan Engine provided by Guido R. Rolon A.; I'm not sure which is which, check your documentation; - added 'dumaru', 'parite', 'gibe' and some other virus names to $viruses_that_fake_sender_re; - add examples to $viruses_that_fake_sender_re to show how to make a default result true, and only list exceptions; - placed a comment in amavisd.conf pointing out the proper syntax for $hdrfrom_notify_* variables; thanks to Wouter de Jong; - bump up the size of $sa_mail_body_size_limit to 150 kB in amavisd.conf - documentation updates; fixed typos and spelling mistakes in the documentation files and in amavisd.conf comments; - new documentation file README.protocol, specifying the new (and the old) protocol between helper programs and amavisd daemon, to be made available with the next major release. The description of the current (traditional) protocol was contributed by Stephane Lentz. --------------------------------------------------------------------------- August 25, 2003 Patch: amavisd-new-20030616-p5 - fix 'Modification of non-creatable array value attempted' bug when no 'Received' header field was present in an infected mail; reported by Paul Miner; - caching of SQL lookups on white/blacklist was based on sender address only, instead of sender _and_ recipient. This could lead to white-/blacklisting of one recipient to affect other recipients of the same message; reported and debugged by Paul Gamble; - added LDFLAGS to helper-progs/Makefile.in. It is needed at least on NetBSD. Patch by Julian C. Dunn (the NetBSD package maintainer for amavisd-new); - more obvious logging of HOLD reason in sendmail/milter setup (reported by Pascal Martinez); - $MAXLEVELS zero or undef should disable the limit according to docs, but was not honoured; reported by Rob Hutton; - if notifications delivery encounters a temporary failure (4xx), propagate this status to the final result instead of only logging a warning; - amavisd.conf: * VirusBuster entry changed to match newer version of the product; information from Marcus Schopen; * another entry in amavisd.conf for KasperskyLab kavscanner (v4.5?), contributed by Simone Marx; --------------------------------------------------------------------------- August 12, 2003 Patch: amavisd-new-20030616-p4 - revert to using alarm() instead of Time::HiRes::alarm(). It is nonstandard to mix the two, and is causing problems on some operating systems (e.g. Solaris); thanks to Geoff Gibbs; - rise log level for log entries on intentional mail drops in case a DSN (a bounce) should be sent, but will not be; such as on rejected bounces, viruses with forged names, and spam from mailing lists. The new log entries now say: '... Not sending DSN ...' and provide more information on the reason for dropping DSN; - if sender is whitelisted, don't insert 'X-Spam-Flag: YES' header field, don't append spam address extension, and don't quarantine. This makes it less surprising, although previous behaviour was according to documentation. NOTE: the documentation still describes former behaviour, this needs to be fixed; - report deaths of command line scanners and some external programs distinctly from normal exits with nonzero exit status; - fix replacing * or {}/* in the pattern with actual file names, causing MkS_Vir (mks32), VFind, Dr.Web Daemon, and KasperskyLab aveclient to receive its file arguments glued together; based on patch by Rafael J. Wysocki; - update 'Panda Antivirus for Linux' entry to work with new (and older) versions of pavcl; updated entry kindly provided by Panda Software; - stop the timer after SMTP transaction is over to better behave in persistent SMTP/LMTP sessions; start the timer at the beginning of a SMTP transaction, in addition to restarting it when DATA mode is entered; - sub mail_via_smtp_single: properly report SMTP response code when all recipients are rejected by MTA, instead of logging a 'mail_via_smtp:' without a value. The problem was commonly seen with the W32/Mimail-A virus which fakes an often invalid local sender address and gets rejected by Postfix outright; reported by Turgut Kalfaoglu; - use SMTP response code 554 (instead of 550) for rejecting syntactically invalid header (according to rfc2476); - add am_id to SMTP response code generated by one_response_for_all() to make it easier for MTA log to be correlated with amavisd-new log; some cosmetic improvements in the generated SMTP response text; - added 'Return-Path:' in notifications to make it more obvious to see envelope address from reports; - the 'Message-ID:' in neutral DSN notifications template was inadvertently pushed into the DSN body; - report undefined spam score in X-Spam-Status header field as 'hits=-' instead of 'hits=0.0' which can be misleading; - indicate blacklisting in X-Spam-Status header field of quarantined messages; - add X-Envelope-From header field to quarantined messages; - added virus names: tanatos, lentin, bridex (alternative names for bugbear, yaha and braid) to the $viruses_that_fake_sender_re; thanks to Harrie Overdijk; - set environment variables LINES and COLUMNS to sensible defaults to avoid some external program get puzzled about the terminal settings (e.g. older versions of pavcl from Panda Software); - another attempt at fixing the Subject header field duplication. The patch amavisd-new-20030314-p2 fixed the case of entirely missing Subject header field, but did not fix the case of Subject header field present but with an empty text. Reported by Steven Cobb and Francois Rolland; - rewritten 'Received' header fields parsing to better cope with valid, as well as with more common cases of broken syntax; used when trying to report originator IP address for believed-to-be-faked senders; - more permissive parsing of SMTP addresses and options on MAIL FROM and RCPT TO commands; - 'neutral' (=space) field in SQL black/whitelists now terminates the lookup search, avoiding fallback to static black/whitelists. It enables recipient to explicitly express its neutral stance towards the sender, overruling the site default; - $sa_mail_body_size_limit now takes into account some portion of the mail header size so that huge mail headers that can cause slow SA calls are avoided (such degenerate cases were reported by Ralf Hildebrandt); - taint fix in read_l10n_templates (as used by the Debian distribution), patch by Henrique de Moraes Holschuh; - don't send recipient notifications to recipients that have bypass_virus_checks/bypass_banned_checks; suggested by Joe Breu; - replace /bin/false with a more usual /usr/bin/false as a last resort exit; - fetch modules 'Net::Ping' and 'bytes', which seem to be needed in certain chrooted setups; - change log level from 0 to 1 for the log entry 'BAD HEADER from'; reported by Thomas Lamy via Debian bug reports; same for 'unrar: all %d members are encrypted'; - fix typo in variable name: $spamassasin_obj -> $spamassassin_obj - explicitly set pipes and sockets to binmode, as is a default since Perl 5.8.1 - documentation updates: * new file LDAP.schema, by Jacques Supcik, PhD * updated README.chroot to tell that /dev/urandom is needed in chroot jail (otherwise creation of MIME notifications fails); thanks to Lynn Duerksen and Jimmy Porter; * updated README.sendmail-dual, thanks to Robert LeBlanc and Stephane Lentz; * added URLs of external programs to INSTALL; * small updates to other doc files; --------------------------------------------------------------------------- June 28, 2003 Patch: amavisd-new-20030616-p3 - avoid 'savemail: cannot save rejected email anywhere' sendmail panic when feeding mail via LMTP and using D_BOUNCE settings in dual-sendmail setup. --------------------------------------------------------------------------- June 27, 2003 Patch: amavisd-new-20030616-p2 - when running chrooted, fix pre-loading of modules needed by SpamAssassin and Razor agents, which SpamAssassin forgets to pre-load by itself; reported by Neil Camara; - updated KasperskyLab AVPDaemonClient entry to protect against carriage returns in collected virus names; thanks to Harrie Overdijk; - updated 'Symantec CommandLineScanner' entry to look also for the new scanner name 'savsecls'; thanks to Guido Rolon; - documented extended uses of read_hash() in the release notes; - updated README.chroot; --------------------------------------------------------------------------- June 24, 2003 Patch: amavisd-new-20030616-p1 - bug fix: allow stderr to be joined with stdout (>&1), instead of creating a file with a name "&1" when calling command-line scanners or doing LHA decoding (it got broken in 20030616); noticed by Henrik Larsson; - bug fix: the X-Spam-Level header field would carry 64 asterisks instead of none when spam is configured to pass (mail tagging only) and when SA was not called (e.g. sender is whitelisted, ...) and tag_level or tag2_level is set to 0 or less; reported by Ralf Hildebrandt; - bug fix: untaint e-mail addresses when forwarding via pipe; reported by Sam Tilders; - modify code to match documentation: $relayhost_is_client should influence $notify_method too, not just $forward_method (adding an extra argument to mail_dispatch() was necessary to make this possible); reported by Zhu Yicun; - SQL failure modes consistency: when initial connect to SQL failed, previous versions of amavisd-new would fall back to static defaults, disabling SQL lookups. Now initial connect failure is fatal, just like if connect failure occurred during operation - mail flow just stops during SQL servers outage, and is resumed when servers become reachable again; - commented-out adding of 'X-Spam-Report' header field to retain behaviour from amavisd-new-20030314 - I find this header too long and intrusive for regular use. Uncomment it if adding X-Spam-Report is desired; - support lzop (.lzo) and freeze (.F) compressors if available; - use external program cpio (if available) to efficiently and safely handle the following archive formats: cpio binary, HPUX binary cpio, cpio crc, old ASCII cpio, new ASCII cpio, and HPUX old ASCII cpio, as well as POSIX.1 tar (also GNU tar) and old tar if allowed (see below) without reading whole archive members into memory; - if your cpio(1L) can read tar format (as is common on FreeBSD and Linux), it is recommended to uncomment the following line (in file 'amavisd'): # /^\.tar$/ && defined $cpio && return do_cpio($part,$tempdir); which will cause tar archives to be safely and efficiently decoded by cpio(1L) instead of Archive::Tar, which loads the whole archive into memory; - use separate file(1) 'classifications' ".uue" and ".hqx" (instead of ".asc") for uuencoded and binhex formats; - merge 'H+BEDV AntiVir' and 'CentralCommand Vexira Antivirus' entries into a single entry - they are basically the same product (the Vexira entry didn't work for new version of Vexira); thanks to Vivek Khera; - do_unrar: some switches to rar/unrar are not recognized by older versions of rar/unrar; issue a more informative log when this happens, and retry without newer switches for compatibility; thanks to Geoff Gibbs; - crop backtrace on SA timeouts at 980 characters; - preload Perl modules Net::DNS::* if spam scanning is enabled; they could not be loaded if running in a chroot jail, so SpamAssassin did not use Razor; - documentation: collected documentation pieces pertaining to LDAP into README.lookups; fix SQL example table data in README.lookups; ( the patched MIME-tools by David F. Skoll is recommended over 5.411, as it better handles broken/bad MIME syntax: http://www.mimedefang.org/ -> Download section ) --------------------------------------------------------------------------- June 16, 2003 amavisd-new-20030616 release notes This is mostly a maintenance release. Only a handful of new features were added that were small or easy enough to provide. - revised the incoming SMTP session abort procedure to properly shut down the remote SMTP client in case of unexpected trouble (e.g. disk full) during SMTP data reception; - subroutine mail_via_smtp_single() overhaul to produce cleaner diagnostics and to properly abort outgoing SMTP session in case timeout occurs in SMTP 'DATA' mode (avoids recipients repeatedly receiving partial message in successive delivery attempts when the receiving MTA responds very slowly); problem reported by Scott Vintinner; - fixed the generation of LMTP response code when D_BOUNCE was chosen, but DSN was not actually sent because sender was thought to be faked or mail came in from a mailing list. Since DSN was not really sent, the final response remained at 5xx (causing bounce by MTA) instead of being converted to 2xx as D_BOUNCE would suggests; reported by Peter Bates; - security: avoid inadvertently untainting values (e.g. mail addresses) at several places throughout the program; - work around a Perl 5.8.0 taint bug where global variable $1 could become tainted, making further untainting attempts unsuccessful; - work around another Perl 5.8.0 taint bug where Encode::encode fills up all available memory when given a tainted string with a non-encodable character. New subroutine safe_encode() provides a wrapper around the Encode::encode; - treat 'Maximum number of files exceeded' the same as 'Exceeded storage quota', i.e. inserting the X-Amavis-Hold header field; noted by Christopher Odenbach; - call exec with explicitly specified program path, preventing exec from even considering calling a shell; - catch and better report the Perl's failure to fork on open; - avoid inappropriately reporting 'Illegal seek' in run_av (and elsewhere) even though nothing went wrong; reported by William Yodlowsky; - new subroutine flatten_and_tidy_dir() recursively descends into a directory containing potentially unsafe files with unpredictable names, soft links, etc., rename each regular nonempty file giving it a generated name, and discard all the rest. - do_unrar and do_unarj now let the archiver itself recursively unpack the archive and then use flatten_and_tidy_dir() to tidy up the result. With previous method it was not possible to extract archive members with names containing non-ASCII characters from rar archives due to a bug in unrar and rar programs. Problem reported by Pan Bambaryla. Protection from mail bombs in do_unrar is retained. - do_unrar: use program rar or unrar, whichever is available. Both rar and unrar recognize the same options to extract. (amavisd.conf: $unrar=['rar','unrar'] ). Under some circumstances unrar falls in a loop while rar extracts the archive correctly. - do_unrar: don't bail out on exit status 1 (warning) from unrar after collecting file names from the archive. Be more careful when extracting file names, such as archive members with names starting with '-' or spaces; Disable showing archive comment and file comments which can easily break member-name parsing. Avoid extracting files in subdirectories twice: once when directory itself is listed, and the second time when each file in a directory is listed. Licence note: unrar is free, rar is not ( see http://www.rarlab.com/ for fresh unrar sources ). - do_unarj: prefer more versatile arj over unarj (amavisd.conf: $unarj=['arj','unarj'] ). Both programs support subcommand 'e' that we use; suggested by Guillem Jover (via Debian support); ARJ since 2.78/3.10 is Open Source (GPL license). When arj is available (recognizing several new archive types, archive versions and options), put these options to good use, recovering several archive members that are unaccessible by the demo program unarj, such as old member versions, memebers with equal names, etc. Disable showing comments. - add X-Virus-Scanned header only if a mail was actually scanned for viruses (i.e. av scan not bypassed for this set of recipients); pointed out by Phil Regnauld; - updated entry for 'KasperskyLab AvpTeamDream' virus scanner; by Daniel Melanchthon; - truncate instead of round the spam level when producing the bar in the X-Spam-Level header field for compatibility with SA. Clip the level bar at 64 characters instead of 60 (more would not be allowed by RFC 2822, although SA used to crop at 100); - subroutine hdr() incorrectly fixed an illegal header field body which did not have a space after a line fold; noticed by Virna Gupta; (the buglet did not show unless one modified the program); - replace several remaining regular expressions /...$/ with /...$(?!\n)/ as a matter of principle (as was done in amavisd-new-20030314-p2 for the more urgent cases). The use of simple $ was almost always subtly semantically wrong, but fortunately in most cases without consequences; - specify open mode explicitly in several 'open's, and protect special characters in file names from being 'too cleverly' interpreted by Perl open (still not all cases done); - SpamAssassin changes umask to 0077 - restore our mask after the SA call; - be a bit more careful regarding platform-independency, distinguishing between use of \012 and \015 against \r and \n (note the \n on Mac is CR); - change log level of several less important av scanner log messages from 2 to 3; - small documentation updates in amavisd,conf; - updated README.lookups: the new substring capture and reference mechanism in regexp lookups explained. The SQL example modified to work under PostgreSQL, thanks to Phil Regnauld; - updated documentation: README.sendmail-dual, README.postfix, README.chroot, new file: README.exim_v4_app2 - by Louis Erickson; NEW FEATURES - provide a command line option 'debug-sa', which is equivalent (but more convenient) then setting $sa_debug to true and starting as '# amavisd foreground'; - list version numbers of the more important Perl modules at startup; - lookup_re(): RE pattern now allows for capturing of parenthesized substrings, which can then be referenced from the result string using the $1, $2, ... notation, as with the Perl m// operator. The number after the $ may be a multi-digit number. To avoid possible ambiguity the ${n} or $(n) form may be used. Substring numbering starts with 1. Nonexistent references evaluate to empty strings. If any substitution is done, the result inherits the taintedness of the key. Keep in mind that the $ character needs to be backslash-quoted in qq() strings (but not in q() strings). Example: $virus_quarantine_to = new_RE( [ qr'^(.*)@example\.com$'i => 'virus-${1}@example.com' ], [ qr'^(.*)(@[^@]*)?$'i => 'virus-${1}${2}' ] ); - read_hash() now returns the hashref (first argument) also as a return value, making it easier to create new hashes (instead of only adding values to existing ones). Also the first argument (hashref) is now optional, and if missing a new hash is created. The following three cases are now equivalent: a) $sa_tag2_level_deflt = {}; read_hash($sa_tag2_level_deflt, '/var/amavis/tag2_levels.dat'); b) $sa_tag2_level_deflt = read_hash({}, '/var/amavis/tag2_levels.dat'); c) $sa_tag2_level_deflt = read_hash('/var/amavis/tag2_levels.dat'); - extend the semantics of the configuration variable $inet_socket_port, which can now either be a scalar as before (i.e. a single port number), or a ref to a list of port numbers, e.g: $inet_socket_port=[10024,10026,10028]; # accept SMTP on all these TCP port When $relayhost_is_client is true, the semantics of $forward_method has changed slightly: instead of taking the port number from $forward_method, it is now calculated as being one higher than the port number on which the incoming SMTP connection came in (one from the $inet_socket_port list). This allows for multiple MTA pairs on the same host to share the same amavisd daemon without having to use multiple loopback interfaces. Useful for example if incoming and outgoing mail is handled by separate mailers on the same host. - add field 'spam_quarantine_to' to SQL table 'policy' - a patch provided by Vivek Khera. For compatibility a missing field defaults to undef, causing lookup search to proceed with the next lookup map; - add fields: spam_lover, banned_files_lover, bad_header_lover, bypass_banned_checks and bypass_header_checks to SQL table 'policy' for completeness; For compatibility a missing field defaults to undef, causing lookup search to proceed with the next lookup map; - add a 'SPAM-TAG, ...' log entry at log level 2, which is produced when inserting spam-related header fields for each cluster of recipients with same settings. It complements the 'SPAM, ...' log entry, which is triggered at kill level. Feel free to adjust their log levels if you think one or the other is redundant; - insert a 'X-Spam-Report' header field at tag2 level when spam is passed. Feel free to comment out the ...append_header('X-Spam-Report'... line if this is undesired. - fold-in the 'bad_headers.patch' to the main code (available in the distribution since amavisd-new-20030314-p1), but renamed *bad_headers_lovers* to *bad_header_lovers* and added $final_bad_header_destiny and *bypass_header_checks*. To refresh memory: Enables checking headers of each mail for invalid (non-encoded) 8-bit characters, and produces a bounce (non-delivery status notification, or a SMTP REJECT if desired) with the full explanation of the problem, with offending header fields trimmed, sanitized and included in the text. New setting $final_bad_header_destiny, defaults to D_PASS for backwards compatibility, but a value D_BOUNCE is suggested. Similarly to other *lovers* settings, a hash and an ACL lookup %bad_header_lovers and \@bad_header_lovers_acl are available, and a setting $warnbannedsender. Certain recipients may be exempt from the checking (*bad_header_lovers*). Similarly mail from mailing lists (Precedence: list or bulk), and mail with null reverse-path mail (e.g. bounces) is passed, even if violating the RFC 2822 header syntax. A log entry is produced nevertheless. Postfix users: this is similar to the Postfix strict_7bit_headers=yes functionality, but produces a much more informative problem report (non-delivery notification) to the sender. One difference is that amavisd-new header check takes into account only the RFC 2822 header, not MIME headers in the mail body. It is reasonably efficient to use amavisd-new as a header checker only, without any anti-virus or anti-spam checks, if desired. - added configuration settings $warnbannedsender and $warnbannedrecip, separating this function from $warnvirussender and $warnvirusrecip, which previously applied to both virus and banned files; - new macro %c which evaluates to spam level/hits (mnemonic: sCore) as provided by SpamAssassin; useful in $log_templ; - new configuration settings *bypass_banned_checks* and *bypass_header_checks*; - new parameter $spam_quarantine_bysender_to (in contrast to the usual $spam_quarantine_to) makes possible collecting quarantined spam by sender names or domains, instead of (or in addition to) the more usual by-recipient map; - new configuration setting $remove_existing_spam_headers (defaults to true); - bring-in the localization (l10n) support contribution from Debian Linux, by Henrique de Moraes Holschuh; see README.l10n, and comments at the call to read_l10n_templates() in file amavisd.conf; - supply X-Envelope-From and X-Envelope-To header fields containing envelope addresses to the messaged passed to SpamAssassin for checking; - there is now an official OID: 1.3.6.1.4.1.15312.2.2 under which LDAP schema for amavisd-new LDAP lookups is to be defined (iso.org.dod.internet.private.enterprise. 'ijs.amavisd-new.ldap' ); --------------------------------------------------------------------------- May 6, 2003 Patch: amavisd-new-20030314-p2 - blacklisted sender only caused spam headers to be inserted, but did not cause rejecting the message and other 'evasive actions' due to over-optimization; reported by Lawrence Farr, Robin Elfrink and Eric Vollmer; - fix improper evaluation of bypass_spam_checks* in multi-recipient mail where some recipients match 'bypass_spam_checks*' and others do not. Spam-check was bypassed if _any_ (instead of _all_) recipient matched 'bypass_spam_checks*'; thanks to Joseph W. Breu; (it got broken in amavisd-new-20030314, the amavisd-new-20021227 was ok); - adding address extensions was not done for *_lovers, only for *_destiny=D_PASS; reported by Steve Khoo. The "deal_with_spam" section needed to be overhauled for this reason. For consistency with the documentation and with the change in adding address extensions, quarantining is now done if mail is considered spam, regardless of it being delivered (e.g. to spam lovers), or not; - if only spam checking is performed, but no virus scanning nor banned file names checking is enabled, and the spam is detected but configured to pass, and the Subject is configured to be edited, the Subject header field was duplicated because the existing Subject header field was not detected. Solution: make MIME entity available even if virus and banned files scanning is not performed. Header information is needed by add_forwarding_header_edits_per_recip() when deciding whether to add or replace the Subject header field in case spam is passed. (workaround for previous versions: don't leave $banned_filename_re undefined. If not needed, set it to an empty list: new_RE() ); - when quarantining to a mbox file, excape 'From ' with '>' only if it follows an empty line, as sendmail mail.local man page prescribes; - added parameter $notify_xmailer_header to control the body of the header field X-Mailer as placed in the notification messages (e.g. bounces) generated by amavisd-new; by default this header is no longer inserted, as it was felt it revealed too much information. Choose from: $notify_xmailer_header = undef; # no X-Mailer inserted (default) $notify_xmailer_header = ''; # X-Mailer: MIME-tools x.xxx (Entity x.xxx) $notify_xmailer_header = 'your-text-here'; # X-Mailer: your-text-here - replace regular expressions /...$/ with /...$(?!\n)/ or with string equality operator in some places (many more to come in the next major release); the use of simple $ was almost always subtly semantically wrong, but in most cases without consequences; - enable timeouts after SMTP connection has been established, to be able to handle cases where MTA is very slow in accepting checked mail, which could result in multiple deliveries if MTA on the input side times out; - debugging aids: call stack backtrace is logged if Mail::SpamAssassin::NoMailAudit::check exceeds allowed time; - LDAP enhancement: allow "hostname" to be a string or a reference to an array, in which case each entry will be tried in order until a connection is made; by Jacques Supcik; - X-Spam-Status now says WHITELISTED if sender was whitelisted; contributed by Joseph W. Breu; - if mail rejection was due to sender blacklisting, say so in the SMTP response; thanks to Robin Elfrink; - updated file bad_headers.patch to apply cleanly against this version. The bad_headers.patch will be folded-in with the next major release, please try it out, comments welcome. For patch description see release notes for amavisd-new-20030314-p1; - specify option 'ExactAddresses=>1' in a call to Net::SMTP::new() in anticipation for the next release of Net::SMTP; - amavisd.conf: updated documentation / comments; - amavisd.conf: removed option '-ni' in the DrWeb virus scanner entry; thanks to Mike Boev and the FreeBSD community ( http://www.freebsd.org/cgi/query-pr.cgi?pr=50893 ) - amavisd.conf: added entry for 'MkS_Vir daemon', thanks to Dariusz Grzegorski; - amavisd.conf: updated entry for 'F-Secure Antivirus' to cope with fsav 4.50 as well as with 4.1x; solution by Juhan Tamsalu; - documentation: updated README.chroot; new README.sendmail describing dual-MTA sendmail setup, small updates to other doc files; March 21, 2003 Patch: amavisd-new-20030314-p1 - fix passing of remote MTA reject code back to the input side (DSN was not sent if MTA rejected the message); - fix broken logic which allowed a virus to pass if a recipient was a 'banned file lover' and the same message also contained a 'banned file'; (or the other way around: pass a banned file if containing a virus and sent to virus lovers); thanks to Ortwin Gentz, I'm glad somebody is looking over my shoulder; - remove a log entry: one_response_for_all: SHOULDN'T HAPPEN The test was inappropriate, but harmless; - enhanced sub run_av() to fix a problem with MkS_Vir virus scanner, where a '*' in the template command was not expanded to actual file names; reported by Pan Bambaryla; - trim virus scanner output (in sub run_av) to a manageable size of 900 characters so as not to clutter log entries and notifications too much; provoked by the Panda pavcl; - move the check for 'Precedence:' mail header field from sub delivery_status_notification() to its caller for flexibility; - replaced the text in notification templates: The message WAS delivered to: into: The message WILL BE delivered to: to reflect the truth. The message may still be rejected by the MTA when forwarding takes place. - small documentations edits/updates; - cosmetic: trim trailing whitespace in the program; - cosmetic: make shorthand subroutine nf() locally scoped; by Jacques Supcik; - make available an experimental patch (file: bad_headers.patch, not applied by default), which, when applied (with patch(1) utility) to amavisd-new, makes it check headers of each mail for invalid (non-encoded) 8-bit characters, and produces a bounce (non-delivery status notification, or a SMTP REJECT if desired) with the full explanation of the problem, with offending header fields trimmed, sanitized and included in the text. Certain recipients may be exempt from the checking (*bad_header_lovers*). Similarly mail from mailing lists (Precedence: list or bulk), and mail with null reverse-path mail (e.g. bounces) is passed, even if violating the RFC 2822 header syntax. A log entry is produced nevertheless. Postfix users: this is similar to the Postfix strict_7bit_headers=yes functionality, but produces a much more informative problem report (non-delivery notification) to the sender. One difference is that amavisd-new header check takes into account only the RFC 2822 header, not MIME headers in the mail body. It is reasonably efficient to use amavisd-new as a header checker only, without any anti-virus or anti-spam checks, if desired. The patch is experimental in the sense that it may change in future versions, but is fully functional when applied; - Jacques Supcik cleaned up and generalized his LDAP lookups support code (which is off by default). In his words: The major change is in the definition of ldap lookups. In the config file, you have to enable ldap first: | $enable_ldap = 1; Then you can define defaults for your ldap queries: | $default_ldap = { | hostname => 'localhost', tls => 0, | base => 'ou=hosting,dc=example,dc=com', scope => 'sub', | query_filter => '(&(objectClass=amavisAccount)(mail=%m))' | }; And then the lookups themselves: | $virus_lovers_ldap = {res_attr => 'amavisVirusLover'}; | $banned_files_lovers_ldap = {res_attr => 'amavisBannedFilesLover'}; | ... With this method, you can define every parameter individually. You could have a different ldap server for each lookup! Like that, the configuration is closer to the one in postfix or courier. The hashes are converted into lookups objects in the Amavis::Lookup::LDAP class method. --------------------------------------------------------------------------- March 14, 2003 amavisd-new-20030314 release notes MAIN NEW FEATURES AT A GLANCE - per-user white- and blacklisting, including a SQL lookup mechanism; refined semantics for white/blacklists; - easier to run in chroot jail, see README.chroot; - can specify two sets of virus scanners: the main list and a backup list, which gets consulted only if all scanners from the main list fail; - split REJECT into separate destiny settings: REJECT and BOUNCE, giving you a choice of who is responsible for sending a non-delivery notifications to the sender; see comments in amavisd.conf; - edited configure file amavisd.conf, providing more informative comments, cleaner examples and simplified common settings; - made it work better with the Unicode-aware Perl 5.8.x (not against it) and distinguish between character data / byte data / octet data (binary), provide UTF-8 (Unicode) support for notification messages, while avoiding some possible nasty surprises in the UTF-8 locale (which is enabled by default in Red Hat 8.0); - SQL SELECT clause settable from the config file; - can store quarantined or forwarded messages as BSMTP files; Changes since amavisd-new-20021227 patch level 2 BUG FIXES - fixed caching problem when bypass_*_checks are used selectively: when only one of the virus/spam tests were performed but the other bypassed due to a per-recipient setting, the result of a non-performed test was inserted into cache as clean. This could cause tests to be bypassed for subsequent message with the same body but different recipients with different bypass* settings, but only within a lifetime of a child process (10 messages by default). Noticed by Jürgen Louis Fluk, thanks! - changed caching of SQL lookups to match the documented behaviour, i.e. SQL lookup result is cached (for the benefit of field lookups) for a single message only, and no longer for the child lifetime. It is possible to revert to the previous behaviour by commenting-out one line (look for $sql->clear_cache). Pointed out by Paul Miner; - see also further down about the field 'banned_files_lover' in the SQL table 'policy', and the UTF-8 (Unicode) workarounds. SECURITY - running amavisd-new in chroot jail is now easier to set up due to added configuration variable $MYHOME and its simplification effects on other settings in the config file. Relevant settings are better commented; - avoid the need to have a shell in chroot jail (use of the new subroutine run_command() instead of Perl functions qx() and system() ); - new file README.chroot with guidelines on how to run amavisd-new in a chroot jail, to prevent possible vulnerabilities from affecting the rest of your system; - added a security notice (web page and other places) that the Unix utility file(1) 3.39 and older contains an exploitable security vulnerability, which can cause system access with access rights of the user running amavisd-new daemon (not root, unless you violated the recommendations). Please upgrade file(1) to 3.41 or newer: ftp://ftp.astron.com/pub/file/file-3.41.tar.gz - new section (Security considerations) on the web page; CLEANUPS AND CHANGES IN amavisd.conf - improve explanation texts (comments) in amavisd.conf, edit the file to improve legibility and structure, change some examples. The daemon is thought to be compatible with amavisd.conf from previous version, but it may be worthwhile to switch to the new file to avoid asking FAQ; - introduce a configuration variable $MYHOME, which is never used directly by amavisd daemon, but is a handy way of setting default values for other configurable variables in the config file. Also makes default settings more chroot-friendly. - introduce a configuration variable $mydomain, which is never used directly by amavisd daemon, but is a handy way of setting default values for other configurable variables when there is only a single domain (with subdomains) behind the mailer. This avoids the need to edit half-a-dozen configuration variables in simple setups; - moved debugging-related settings to a separate section (Section VIII - Debugging); - make SQL SELECT clause settable from the config file; - renamed variable @local_domains to @local_domains_acl for consistency; The old name became a synonym for @local_domains_acl, so compatibility with existing config files is retained; - comment out the example: read_hash(\%whitelist_sender, ...); it slipped into the amavisd-new-20021227 released code from my tests; - comment out the setting $relayhost_is_client=1; It is only useful for multi-access daemon setup, and potentially confusing for others; VIRUS SCANNERS - introduced a list of backup virus scanners: @av_scanners_backup, complementing the normal list of virus scanners. If no virus scanners from the @av_scanners list produce 'clean' nor 'infected' status (e.g. they all fail to run or the list is empty), then all scanners in the @av_scanners_backup list are tried. When there are both daemonized and command-line scanners available, it is customary to place slower command-line scanners in the @av_scanners_backup list; - removed the Mac and SafeMacDfHandling options from SAVI-Perl, as they are no longer supported by Sophos library since 2003-03; set new option NamespaceSupport to SOPHOS_DOS_AND_MAC_FILES; thanks to Paul B. Henson; - changed a default to initialize SAVI-Perl only at startup time, and no longer every time the child process is started. This is because starting with SAVI 3.0 (March 2003) the initialization is slower. NOTE: it is now necessary to restart amavisd master process after SAVI database is updated, to make it notice new virus descriptions. - in 'MkS_Vir for Linux' entry replaced options -e -c with -s, to make mks work, suggested by Pan Bambaryla; - added status 8 ('corrupted') to the list of clean statuses for 'KasperskyLab AVPDaemonClient', as is already the case for some other virus scanners. See amavisd-new web page (Security considerations) if the idea bothers you; - RAV command line scanner no longer supported off-the-shelf, as interfacing to it seems to be violating their license terms. Btw, if RAV command line scanner does not work as user amavis, check permissions and ownerships of /usr/local/rav8/* or reinstall RAV as the user that will run amavisd; thanks to Oyku Gencay; - RAV does not like 'new ask_daemon same as Amavis::AV::ask_daemon sophos_savi same as Amavis::AV::sophos_savi amavisd.conf file is changed to use the short names, but full subroutine names are still valid. - Provide protection mechanism for SMTP/LMTP server code to limit the number of recipients in a mail transaction to some sane value ($smtpd_recipient_limit, default 1000). Even if the number is exceeded, mail will still be normally delivered in more than one transaction (provided your MTA is rfc2821 compliant). - SMTP client code now correctly handles the situation where remote SMTP server returns '452 Too many recipients'. When this happens the mail forwarding or submission is split into several SMTP transactions, and as long as the remote SMTP server is willing to accept at least one recipient per transaction, mail delivery is assured. - Auto white-listing (AWL) by SpamAssassin may be turned on by setting $sa_auto_whitelist to true (contributed by Hamish Marson). This only works as one global database, the per-recipient AWL is presently not available. - New configuration variables $hdrfrom_notify_sender, $hdrfrom_notify_admin, and $hdrfrom_notify_spamadmin, to remove the overloading (double semantics) from variables $mailfrom_notify_*. Now $mailfrom_notify_* control ONLY envelope addresses, while $hdrfrom_notify_* only controls From header in notifications. The default values for $hdrfrom_notify_* are chosen to preserve compatibility. - The spam headers in quarantined message now reflect the lowest tag and kill levels when more than one recipient is given, instead of reporting the levels for the first specified recipient. - Tested with SpamAssassin 2.50 and with its new Bayesian classification (used as a global database by amavisd-new). Given two sets of spam and non-spam (=ham) messages, one can build a database by using new SpamAssassin utilities sa-learn-spam and sa-learn-nonspam as user amavis, and the database built this way will be used by amavisd-new/SpamAssassin. - Initialize SpamAssassin as $daemon_user (no longer as root), as a safety measure, and to ensure the files it creates are still accessible after the UID change. Incompatible changes to 20021116: - Different parameters in @av_scanners list for daemonized antivirus scanners (as described above). - lookup_hash: an undefined hash value was previously treated as a special case, resulting in value true. This is no longer the case, which brings it in line with other lookup methods. It is not likely anyone will be affected, as this practice was only documented but not encouraged by any example. --------------------------------------------------------------------------- Here is a brief description of patches that needed to be applied to amavisd-new-20021116, and were available from the amavisd-new web page: patch5 (2002-12-09): - fix a simple test to decide if the anti-virus and anti-spam code is needed; - use the /m regexp modifier by default when parsing av-scanner output for virus names - seems like (?m) in qr is ignored (have we found a Perl bug?); - fix DSN to report original smtp response in case reporting MTA is remote (thanks to Chris Hastie for his keen eye in understanding DSN); - fix reporting quarantine mailbox name(s) in case of per-recip quarantine or quarantine forwarding to MTA; - macro %q (quarantine id) is now a list of quarantine mailbox names or addresses (may have more than one entry in case of per-user quarantines); - macro %v (av-scanner output) is now a list of scanner output lines, to facilitate formatting of multi-line av-scanner output; - fix spam admin notifications, which used the virus template instead of the spam template; - fix problem of undefined entity causing spam notification failure in case other content checking (virus/banned) is not enabled; - make SQL lookups (DBI access) optional - no DBI code is loaded if @lookup_sql_dsn of empty; - support SAVI-Perl module 0.10 or later (direct access to the Sophos library engine) ( http://www.csupomona.edu/~henson/www/projects/SAVI-Perl/ ); - add macro %j that evaluates to message 'Subject:' header field; - fix minor SMTP protocol inconsistency when multiple MAIL FROM commands are issued by the client; - more informative timing report when more than one av-scanner is used; - make sendmail initial submission happy when null reverse-path is specified (instead of the correct empty argument, which works but makes sendmail issue a warning, we now use <> in arguments to specify null paths), (thanks to Sebastian Hagedorn and Dale Perkel); - replace log entry "spam from=<%s>, to=%s" with "SPAM, <%s> -> %s ...". - dropped (redundant) historical %bypass_checks @bypass_checks_acl (use bypass_virus_checks and %bypass_spam_checks instead); patch4 (2002-11-21): In amavisd: - SQL lookup fix; - finished per-recipient header edits and split-forwarding, where one multi-recipient message may be forwarded as several messages if different headers need to be inserted for different recipients (not with milter); - the check for banned file contents now looks for both the short and the original full type classification as provide by 'file' utility (thanks to Wolfgang Lumpp for the idea); - #!/usr/local/bin/perl => #!/usr/bin/perl - cosmetics In amavisd.conf: - suggested group name change $daemon_group 'sweep'=>'amavis'; so that non-Sophos users are not puzzled where 'sweep' group comes from; - added path and fixed args for KasperskyLab AVPDaemonClient (thanks to Mike Hall); - added path for Trend Micro FileScanner vscan (thanks to Eduardas Paulavicius); - added 'use strict;' and '1;' at the end; prevent read_config from failing if the config file does not return true (thanks to Vivek Khera); - cosmetics, fixed typos in comments; patch2 & patch3 (2002-11-18, 2002-11-19): - fixes a problem with 'recip_done' object method (causing TEMPFAIL) when $warnvirusrecip is enabled (thanks to Chris Hastie, Dale Perkel, and Didi Rieder); - pass null return path to sendmail in such a way to make it happy (thanks to Sebastian Hagedorn and Dale Perkel); - added a safety precaution to prevent somebody shooting himself in the foot by specifying an empty $forward_method while using the SMTP in/out setup (e.g. Postfix), which could send mail down the bit bucket; patch1 (2002-11-17): - fixes a problem in milter setup where per-recipient response codes needed to be handled differently (the problem was causing a TEMPFAIL) (thanks to Stephane Lentz and Didi Rieder for the initial milter testing and problem reports); --------------------------------------------------------------------------- November 16, 2002 amavisd-new-20021116 release notes - provide a mechanism to load only required code sections (anti-virus scanning, anti-spam scanning, SMTP/LMTP server module, traditional amavis client & milter server module), resulting in reduced memory usage and less installation dependencies; - introduce the per-recipient status handling and make possible some per-recipient functionality that was not possible before, such as handling the per-recipient spam thresholds. This required a major rewrite and cleanup of some sections, hopefully providing cleaner mechanism for possible future advances in this area. Consider for example a previously incorrectly handled situation where there is more than one recipient, and some (but not all) of them get a reject from the (outgoing) relay MTA - it is not possible for a single SMTP response or status code to describe the situation; - generate proper (non-)delivery status notifications (DSN), compliant with rfc1892 (now rfc3462) and rfc1894 (now rfc3464). This form supersedes the simple virus/spam sender notifications, but also covers the area of genuine delivery failures, such as selective recipient rejects by the outgoing relay MTA, which could previously lead to multiple deliveries; NOTE: to avoid sender getting two non-delivery messages (one from MTA, and another from amavisd), do not set $warnvirussender and $final_virus_destiny=-1 (REJECT) at the same time (and equivalently for spam settings); - enhanced mail system status codes (rfc1893 (now rfc3463), and rfc2034) are now included with all SMTP responses and DSN notifications; - added checking for banned MIME types and names. If any mail part matches, the whole mail is rejected, much like the way viruses are handled. A list in object $banned_filename_re can be defined to provide a list of Perl regular expressions to be matched against each part's: * Content-Type value (both declared and effective mime-type), including the possible security risk content types message/partial and message/external-body, as specified by rfc2046; * declared (recommended) file names as specified by MIME subfields Content-Disposition.filename and Content-Type.name, both in their raw (encoded) form and in rfc2047-decoded form if applicable; * file content type as guessed by 'file' utility and classified into short type names such as .asc, .txt, .html, .doc, .jpg, .pdf, .zip, .exe, ... - see subroutine determine_file_types(). This step is done only if $bypass_decode_parts is not set. NOTE: by default the $final_banned_destiny is set to 1 (pass), so detected banned file names only cause a header line to be added, quarantining, and added address extension - but the mail is delivered nevertheless. This default is set so that we can get initial experience without being too obtrusive. Change as you prefer. - besides SMTP/ESMTP protocol, the server now also accepts LMTP protocol (rfc2033). This now explains why I had to implement PIPELINING, ENHANCEDSTATUSCODES and 8BITMIME SMTP extensions, as these are required by rfc2033. One advantage of using LMTP to feed content filter is that LMTP uses per-recipient status response, as opposed to 'one-size-fits-all' SMTP status response, which require that the SMTP client (e.g. content filter) is capable of generating DSN. Another advantage is specific to the Postfix setup, as Postfix LMTP client supports multiple transactions per session, saving on connection teardown/reconnect for every message being checked. To enable LMTP feed from Postfix, add 'max_use=10' to main.cf, and replace last argument 'smtp' with 'lmtp' in the master.cf line: 'smtp-amavis unix - - n - 2 lmtp'. No changes are needed in amavisd-new, both protocols can coexist, the distinction is based on HELO/EHLO vs. LHLO command. Btw, older versions of Postfix lmtp client inappropriately lowercased the envelope addresses. This is fixed in Postfix Snapshot 1.1.11-20021015, and in the regular Postfix release 1.2 (when available). Lowercasing can be a problem for addresses where local part is case-sensitive, although such setups are rare. - improved per-recipient SMTP response code handling when sending mail via SMTP. Previously one rejected recipient (by MTA) in a multi-recipient message caused the whole message to me rejected. This is now correctly handled with the new ability to send DSN; - added command line option '-c config-file' so one can override the default location of the configuration file (/etc/amavisd.conf); - explicitly set PATH and HOME environment variables (settings: $path and $helpers_home) - added another form of lookups: Perl regular expression matching. See README.lookups for details. Corresponding new variables in amavisd.conf are: $virus_lovers_re, $spam_lovers_re, $bypass_virus_checks_re, $bypass_spam_checks_re, $local_domains_re, as well as $banned_filename_re, $viruses_that_fake_sender_re, $keep_decoded_original_re, $whitelist_sender_re, $blacklist_sender_re. - besides whitelist_sender* lookup tables, there is now also a blacklist_sender* set of tables, which causes mail to be declared spam and to skip remaining spam checks. See examples in amavisd.conf. - provide a configurable lookup table $keep_decoded_original_re of file types, for cases where unpacker is not very trustworthy. The lookup key is what 'file' utility returned. If the part contents match the lookup table, we keep both the original and the unpacked file for virus scanner to check; - provide a configurable list of regular expressions $viruses_that_fake_sender_re, which recognizes viruses that usually fake envelope sender address. Don't send sender notification if a match is found. For syntax of the new regexp lookup tables see README.lookups; - specifying per-recipient quarantine address or location is now possible by setting $virus_quarantine_to and $spam_quarantine_to be a ref to a hash lookup table. Thanks to Vivek Khera for the idea; - no 'configure' and 'make' for the daemon; all configuration is done via amavisd.conf configuration file at the daemon startup time; Also the ./mta, ./av and ./notify configuration subdirectories are now gone; - provided a sample init shell script amavisd_init.sh (edit to will, and move it to /etc/init.d/amavisd if you want). Based on the script from Wil Cooley; - one-shot debugging mechanism: if envelope sender matches @debug_sender_acl lookup table, turn debugging fully up just for this one message and cause temp file and directories not to be cleaned for this message. This facilitates debugging a particular problem even in the presence of regular traffic; - cleaner debug log entries for multi-transaction SMTP/LMTP protocol sessions; - cleaner log entries - avoid misleading and incomplete text when quarantine is disabled; thanks to Michael Leone; - avoid using (nonstandard) field width in the %e format specifier when calling strftime, which lead to bad date syntax on some systems; - local time zone offset is now automatically computed, no more '-0000' in rfc2822 dates; - put Message-ID field value in angle brackets as required by rfc2822 when generating notifications; - add a In-Reply-To field to notifications when original Message-ID is known; - don't bounce a virus (or spam) back to a mailing lists even if $final_virus_destiny (or $final_spam_destiny) is set to REJECT; a patch by Brian May from the Debian support crew, thanks; - because some external module may play games with STDIN and STDOUT (like SpamAssassin seems to do when local_tests_only=>0) run the input protocol directly on the Net::Server's socket, not on STDIN and STDOUT, which are aliases to the socket; - a patch to amavis-milter.c by Didi Rieder to support REJECTing mail, by instructing sendmail to return non-delivery notification to sender; - a modified amavis.c helper program to make possible to invoke local delivery agent from it, for those still using such a setup; - a new macro %l is available for use in notification messages (via 'expand'); it evaluates to true (1) if the sender matches @local_domain, and returns empty otherwise; by default it is now used in creating a 'Subject:' line, inserting word 'LOCAL' before the sender name when appropriate; - new macros %D and %N expand to lists of recipients that got the mail delivered (%D), or not-delivered (%N). Union of both sets gives %R, i.e. a list of all recipients as specified in the envelope; - dropped macro %a - improved parsing of e-mail addresses according to rfc2821 (full address literals syntax, etc.); - proper line folding for generated rfc2822 header fields; - proper SMTP response wrapping (for very long responses such as the ones that include a trouble report) according to rfc2821; - allow for obsolete rfc822 syntax of permitting whitespace before colon in header field name; - added optional spam-sender nondelivery notifications, based on patch from Lazslo E. Miranda (lazslo@dcc.ufmg.br) and Fernando F. Morais (frota@cecom.ufmg.br); - dynamically change process name (Perl variable $0) to reflect the process state; suggested by Chip Paswater. Not all operating systems make this process state visible by ps(1); - determine location of external programs (or their absence) at startup time; in amavisd.conf one may specify absolute path or just rely on PATH. This mechanism is also used to determine absolute path of the daemon itself, making reload (after HUP) more predictable; - explicitly specify lock file for serialization to be used by the Net::Server::PreForkSimple module, instead of relying on default provided by POSIX::tmpnam. The default approach has a possibility for a minor security problem, because the lockfile is created with open()..., so it will follow symlinks. Observed, and a patch provided, by Jarno Huuskonen; - new defense against mail bombs: for the cumulative total of all decoded mail parts we set max storage size. The formula is: quota = max($MIN_EXPANSION_QUOTA, $mail_size*$MIN_EXPANSION_FACTOR, min($MAX_EXPANSION_QUOTA, $mail_size*$MAX_EXPANSION_FACTOR)) In plain words (later condition overrules previous ones): allow MAX_EXPANSION_FACTOR times initial mail size, but not more than MAX_EXPANSION_QUOTA, but not less than MIN_EXPANSION_FACTOR times initial mail size, but never less than MIN_EXPANSION_QUOTA - if the permitted quota is exceeded (or the defense triggered by other similar safeguards), the virus scanning is skipped to protect the virus scanner from tripping over the mail bomb, a header field is inserted: X-Amavis-Hold: ... reason ... and a log entry 'Placing on HOLD: reason' at level 0 is produced. Also the temporary directory is preserved. The 'X-Amavis-Hold: ...' header field can be used by your MTA to put the message 'on hold' (freeze). If MTA is not set up to catch messages with this tag (the default), they are passed normally to recipients. This is likely to be the best action under the circumstances. - tested with razor-agents-2.20 and SpamAssassin-2.43 (Razor2 is now called by SpamAssassin, and no longer directly by amavisd-new). New configuration variable $helpers_home, which defaults to $TEMPBASE. Thanks for hints by Chris Hastie and John Stewart; - to avoid CPU loop in SpamAssassin-2.43 and earlier, my patch needs to applied to SA - see amavisd-new web page; - to avoid taint problem in Razor 2.20 (if SpamAssassin-2.43 is configured to call it), my patch needs to be applied to it - see amavisd-new web page; - chroot available (but not well tested): $daemon_chroot_dir = '/var/amavis' - provide a fail-over mechanism for SQL database connect - given a list of SQL servers/databases, pick the first that is available. Thanks to Ken McKittrick for making available the patch, and to Ben Ransford for writing it. - remove existing virus-related and spam-related headers (some of them optionally) if we'll be providing our own; suggested by Borut Mrak; - avoid 'insecure dependency' in lookup_sql when calling DBI::execute (thanks to ric* at mpc.com.br) - fixed macro %H to provide original header lines, not the ones stored internally by MIME::Entity, which may have been modified; (thanks to Chris Hastie for noticing the problem); - no longer inserts X-Razor-id header field; INCOMPATIBLE CHANGES: - when specifying boolean values to variables in the amavisd.conf file, please specify 1 (or old style "yes") for true, and 0 or "" or undef for false. The old style "no" yields true for Perl, and is only still supported (converted to 0) for some traditional variables for compatibility with amavisd(-snapshot); - removed variables $sendmail_wrapper*, and changed the syntax for specifying $forward_methor/$notify_method, with the intention to do all mail sending settings at one place. See amavisd.conf for examples; - variable $mailto is now deprecated (but still works as a fallback default for compatibility with previous version). Use $virus_admin and $spam_admin lookup tables instead, they also offer a mechanism to specify per-domain administrator address; - variable $warnadmin is no longer used. Use $virus_admin and $spam_admin lookup tables instead. Not specifying administrator address (e.g. leaving $virus_admin, $spam_admin and $mailto undefined) turns off admin warnings. - dropped variable $LOGDIR, the variable $LOGFILE now specifies the full path; - renamed: $warnsender -> $warnvirussender, $warnrecip -> $warnvirusrecip - dropped variables $enable_relay, $sendmail_cf_orig, $QMAILDIR; they were never used in amavisd-new; - dropped $MAX_ARCHIVE_NESTING, which is replaced by new storage limitations: $MIN_EXPANSION_QUOTA, $MIN_EXPANSION_FACTOR $MAX_EXPANSION_QUOTA, $MAX_EXPANSION_FACTOR - SQL database is now specified differently (@lookup_sql_dsn) MTA-SPECIFIC: sendmail/milter - $notify_method now specifies deferred delivery mode ('-odd') by default, when submitting notifications to sendmail. This is to avoid calling milter immediately during submission, which in turn calls amavisd-new, possibly leading to a deadlock situation when the number of amavisd-new child processes is small. Seems like this change is needed since Sendmail 8.12 or so. The following recommendation is from mimedefang-filter man page: | You MUST run a client-submission queue processor if you use | Sendmail 8.12. We recommend executing this command as part of the | Sendmail startup sequence: | sendmail -Ac -qp1m DECODERS-UNPACKERS: - rewritten decode_parts() to allow for retaining source text if the unpacker sw is considered unreliable. This more cleanly resolves the problem reported on the amavis-user mailing list on 2002-06-06: "Amavisd passing through VBS/VBSWG.gen@MM" A nice side-product is that a directory search is avoided for each nested unpacking step; - refined do_unzip to control and limit the size of decompressed members (among others it defends against the 42.zip-type bombs); - rewritten external decompressors interface to the gzip/bzip2/compress family. Instead of using a 'system' call, they are now called through fh_copy, making possible to control and limit the size of the decompressed contents on the fly, avoiding denial-of-service attacks. Affected: b(un)zip2, g(un)zip, (un)compress; - fh_copy now uses IO::Handle object to assure the forked process gets reclaimed even in case of aborted contents extraction; stdin gets redirected to /dev/null or to a specified input file for the exec'd process; - Convert::UUlib is called again (the amavisd-new-20020630 removed its usage due to problems with improper decoding). This time the originals are kept, so that virus checker sees both the original and the attempted-decoded part; - completely rewritten do_ascii and its usage of Convert::UUlib to fix: * coding error (bitwise op treated as logical op and improperly negated): ... if (!$uu->state || !FILE_OK || -z $newpart); * it never reset the state, so if a successfully decoded ASCII file contained another ASCII file, each decoding level would decode all previously decoded parts again, plus add new ones at this level; recursion would not stop until the hard limit, resulting in TEMPFAIL; * it never checked nor reported errors that should have been detected (I/O errors, out of memory, trouble accessing or creating files); * more informative log entry; - save MIME preambles and epilogues (if nonempty) as extra (pseudo)parts to be scanned. This also mitigates the problem of syntactically-incorrect MIME mail as produced by some user agents, which (rightfully) gets treated as one long preamble by MIME-Tools, and previously went by unchecked; - supports unpacking arc archives using 'nomarch' (by Russell Marks, http://rus.members.beeb.net/nomarch.html), (thanks to David D. Kilzer for the initial code). Using 'nomatch' fixes a nasty habit of arc which gratuitously appends a form-feed at the end of file when using the 'p' (pipe) option, which might mislead a virus scanner. Besides, 'nomarch' is GPL licensed; - fixed a 'broken pipe' problem when calling unrar, thanks to Ricardo Campos Passanezi and Rainer Link; VIRUS SCANNERS: - rewritten interfacing for most command-line virus scanners. All settings for them is now done in amavisd.conf. New ones may be added without having to modify the daemon source. More complex scanners (e.g. daemonized scanners) still need to have a corresponding interface routine in the daemon; - rewritten Sophie and Trophie interfaces to be more resilient to Sophie/Trophie daemon restarts during virus database reloads, avoiding an unnecessary retry (TEMPFAIL) - thanks to Cor Bosman for the suggestion and code, and to Dale Perkel for testing the Trophie interface; - make sophos sweep tolerant to encrypted attachments: if all files are password protected, then the scanner failure is ignored and the message is allowed to pass. Based on patch by Radu Greab; - updated nai uvscan interface to recognize the result 'Found trojan or variant Exploit-CodeBase !!!', a patch by Anton Berezin; - Clam Antivirus supported; - fixed a problem in 'avp' scanner interface ($TEMPBASE not imported), thanks to Joshua E Warchol; - avpdc (KasperskyLab AVPDaemonClient): recognize additional exit codes, based on patch from Christian Hammers; - Panda new regexps, no TERM vt100 setting (thanks to Benjamin Zwittnig); - Trend vscan exit code seems to be the number of infected parts. Updated the test to reflect that, based on observation from Stephane Lentz; - MksVir scanner interface returns error code 2 if viruses are removed (if --clean option is passed) - a patch by Robert Litwiniec; ---------------------------------------------------------------------------- June 30, 2002 amavisd-new-20020630 release notes Since it seems like several people are adapting amavisd-new in details to their requirements, and certain improvement requests have much in common, I'm making available the 20020630 release, shortly before leaving on vacation. As this means my support will be absent for the coming few weeks, and this release is perhaps by few days premature, please consider it primarily a development and new features release. It is available at the usual location at: http://www.ijs.si/software/amavisd/ (or ask Google about 'amavisd-new') Having said that, it is still a fully functional and tested version, and it is running in production at a couple of sites now. Also it is my version of choice in view of dependability, having it running at our site while I'm away. If you get into trouble, you can still go back to amavisd-new-20020517 with which it is fully upwards compatible. The main changes and features since amavisd-new-20020517 are: - the code is thoroughly rearranged, interfaces cleaned, separated into namespaces (packages), several sections generalized (e.g. lookups, appending/editing header lines). The AV scanner and unpacking sections are still mostly the same and compatible with amavisd, so whatever improvements and new AV scanner support becomes available for amavisd, applies almost without a change to amavisd-new; - SMTP on the input side (used with Postfix and Exim) now talks ESMTP (rfc2821) and not just rfc821, including some SMTP extensions: command pipelining (rfc2920), message size declarations (rfc1870), and 8bit-MIME transport (rfc1652). The main reason for this was the change in recent Postfix versions which can now do MIME transformations to support 7bit transports (implied by SMTP). To ensure the transparent 8bit path and avoid message transformation by MTA, amavisd-new needs to declare it does present an 8bit-clean path. A side benefit is a little speedup in passing chunks of mailing list addresses due to pipelining support; - split certain previous amavisd.conf settings (variables) into several variables or lookup tables: * sender address for notifications: $mailfrom -> $mailfrom_notify_admin, $mailfrom_notify_sender, $mailfrom_notify_recip, $mailfrom_notify_spamadmin (these may also be empty to specify null reverse path <>, which is most useful for sender notification); * administrator address for notifications: $mailto -> $virus_admin, $spam_admin (per-sender lookups) * %bypass_checks -> %bypass_virus_checks, %bypass_spam_checks - supports SQL database lookups via Perl module DBI (interface to popular database types). Some examples are provided, if you need other SQL lookups just modify the code by analogy. SQL lookups are most useful for per-user settings of virus_lovers, bypass_virus_checks, bypass_spam_checks, spam thresholds etc. when the user base is large and subgroups can not be identified through their (sub)domains. Another use is for dynamically changing settings without having to restart amavisd-new; - can optionally insert 'Received:' header if acting as a mail relay (not with milter); does loop detection as required by rfc2821 section 6.2; - notification messages now contain 'Date:' and 'Message-id:' headers; - quarantined viruses contain X-AMaViS-Alert header line with names of detected viruses; - quarantined spam contains X-Spam-Status and X-Spam-Level header lines; - optionally send spam admin notifications, which include the full SpamAssassin spam report and message header; - when started as root, changes UID and GID to $daemon_user, $daemon_group; - to facilitate startup scripts and debugging, supports few simple command line parameters: amavisd ... standard run: changes uid/gid and daemonizes amavisd start ... same thing amavisd debug ... starts with full debug level, stays attached amavisd reload ... finds amavisd master process and sends it a HUP amavisd stop ... finds amavisd master process and sends it a TERM - some more (minor) configurable options: $daemonize, $pid_file, $replace_existing_extension, $localpart_is_case_sensitive - no longer calls Convert::UUlib for uuencoded, xxencoded, and binhex attachments. The first two are handled by MIME::Parser, the remaining are likely to be handled by anti-virus scanners, especially when some virus would use such encoding. The use of Convert::UUlib is dropped because it was causing recent problems with garbling virus so that it could no longer be recognized by AV scanners, and because the underlying library does not seem very dependable. See the thread 'Amavisd passing through VBS/VBSWG.gen@MM' from the beginning of June 2002 in amavis-user mailing list archives; - quarantine files now include internal amavis id in the file name, instead of the process number; previous naming scheme could stumble across a name contention on a busy system; - only a recommendation: logging via syslog is now preferred to direct logging to a file. It serializes the logging, and avoids locking/unlocking and reopening a log file by amavisd* for every log entry. The syslog daemon does it more efficiently and reliably. NOTE for Linux users: make sure you prefix the file name in syslogd.conf with a '-' tag to disable fsync after every write; this is most necessary for heavy logging such as from MTA and/or amavisd-new. The downside is that you may lose the last few log entries in case of machine crash. See man page of your syslogd for details. - changed examples according to rfc2606 and recent complaints on the postfix-users list. CAVEATS: - header rewriting is only available in SMTP-in/SMTP-out setup, i.e. with Postfix and Exim, but not with sendmail milter setup; - the SQL lookups are a very recent addition and not so well tested as the rest of the program. Also since SQL lookups are supposed to introduce per-user settings (e.g. spam thresholds), the code still does not accommodate it, and spam thresholds of the first recipient in a message affects the whole message; - as Razor2 is still pretty much unstable, I left the spam_scan() routine much as it was in the May version. Contributions welcome; - as I ran out of my time for this release, I didn't prepare a separate version without SpamAssassin and Razor 1.20 support, so you will need to strip it out if you do not need it. Similarly for the required DBI Perl module. I'm very sorry; - the time stamp used in the 'Date:' header in notifications, and in an optional 'Received:' header, does not contain true time zone offset, but -0000 instead (standard meaning for unknown). I did not want to include a fat Perl package for handling time zones. For now just edit the subroutine rfc2822_timestamp() if you want to change that; - amavisd.conf settings $daemon_user and $daemon_group are not automatically set by ./configure options. Please set them manually. I would like to thank many people on the amavis-user list, on the postfix-users list, and in private conversations, who contributed valuable ideas and improvements, and offered much appreciated encouragements. --------------------------------------------------------------------------- May 17, 2002 Available at: http://www.ijs.si/software/amavisd/ amavisd-new-20020517 is primarily a response to popular demand for Mail::SpamAssassin support. If amavisd-new-20020424 meets your needs, there is no urgent need to upgrade. There are no incompatible changes between these two versions, except some new (optional) amavisd.conf variables, so you may keep old amavisd.conf file if you wish. A new file README.exim is now provided, so that Exim 4.x is now a supported and tested configuration, besides the usual Postfix and sendmail/milter MTAs. Also included is a brand new qmail amavis client by Lars Hecking (untested, please try it), plus his updated version of amavis.c (no longer needed in recommended Postfix and Exim configurations, and Sendmail milter and qmail configurations use a different client). Several files are unchanged from the base amavisd CVS release 2002-05-13 and do not reflect the amavisd-new state: FAQ, HINTS, INSTALL, BUGS, NEWS, TODO, ChangeLog, tests/, doc/. Please start with this file README.amavisd-new-RELNOTES and follow it. More recent instructions and last-minute changes are available from the web page. Changes since amavisd-new-20020424: - supports Mail::SpamAssassin and Vipul's Razor (1.20 required) for spam checking (but not for modifying mail body - only add headers and/or address extension, or reject/discard/quarantine spam). NOTE: spam checks are off in the default amavisd setup. Copy file ./amavis/amavisd.in.all to ./amavis/amavisd.in before running ./configure and make, to use the spam-check-enabled version! Amavisd calls Mail::SpamAssassin directly, avoiding the need to set up spamc/spamd or to chain filters. This is more efficient, one daemon less to worry about, although maybe less flexible for some taste. At the moment the SA per-user database is not used. Feel free to experiment with it and let me know what you came up with. The usual SA config files are observed, but remote tests are disabled by default (Razor is handled directly, RBL lookups can/should be performed early by MTA which knows what IP address mail came from, and that information is (mostly) lost afterwards). If you feel otherwise, change the hard-wired settings in the call: $spamassasin_obj = Mail::SpamAssassin->new( { dont_copy_prefs => 1, local_tests_only => 1 } ) (also the Razor score contribution is hard-wired, modify: '$razor_spam_found ? 3 : 0' to will). Amavisd-new handles Razor checks directly to be able to exercise more control over it than would otherwise be available through SA: timing, signatures are needed for insertion into header, skips one-liner body checks which are common Razor false-positives. To avoid SA calling Razor again, either keep the default setting 'local_tests_only=>1', or set 'score RAZOR_CHECK 0' in the SA configuration. Added are 'whitelist_sender' hash and ACL lookups (see README.lookups), which approves spam from specified SMTP originator addresses - SpamAssassin can only check and whitelist rfc822 headers, not the envelope addresses, and I see no way of passing envelope addresses to it - which is a pity, as important information is lost. SpamAssassin checks are computationally quite expensive compared to other amavisd activities, and the time needed for SA check goes up significantly with the message size. SA check are skipped (but not the Razor check) if mail size exceeds 64k. My analysis shows that presently less than 1% of spam exceeds 64k characters, and this is probably well below the false-negative SA rate, so it is not worth wasting time to check large mail. remaining changes: - new file README.exim (thanks to Jochen Erwied, Patrice Fournier and Igor D'Astolfo); - updated README.postfix to describe how to avoid running header_checks, body_checks and dns_lookups in Postfix twice; - put back file README.customize, which was omitted from amavisd-new-20020224 by mistake (but available with previous version and on the web); - added introductory paragraph to README, the rest still needs to be updated; - added missing last-step check for '.' in hash lookups to match examples and make it more useful; - added timeouts to certain tasks to make it better suitable for unattended operation (less, but still subject to certain DoS attacks, similar to the official amavisd). The main reason for adding timeouts is that we don't want spam checking to slow down amavisd operation too much: if it takes too long, just skip it (assume not spam) and move on; - change unmangle_sender() to believe sender address for Klez viruses; this is sometimes wrong, but seems like people prefer to sacrifice few false accusations in favour of some warranted sender notifications. Feel free to hack this routine at will (and publish good ideas), it is intended to be modified; - use Perl module Errno instead of errno.ph to avoid using broken file errno.ph on some popular platforms; - make possible to have per-sender-domain administrator e-mail address for admin notifications (hash lookup %mailto, sub warn_admin() ) - relax temporary file/dir protection to allow them to be readable by the group. This makes possible for anti-virus (daemon) checker to run under a different user (but in the same group). Doing so should be safer as it makes impossible for virus-checking daemon to clobber files. It is only supported in the all-SMTP configuration. To use it with traditional amavis clients (e.g. milter), you will have to modify their sources to change umask and mods for file/dir they create; - avoid changing sender address <> to <""> under certain circumstances (not strictly wrong, but still a bad idea); - replaced header 'X-Razor-Warning:' with SA-compatible 'X-Spam-Status:' ; the 'X-Razor-id:' is still provided to facilitate user spam reporting; - added '--' between options and argument when calling $sendmail_wrapper to be triply sure we avoid problems with some mailers (note that $sendmail_wrapper is only still needed in the sendmail setup, Postfix and Exim do not need it in the recommended configuration); - tidy the %local_delivery_aliases mechanism for local delivery / quarantine; - allow MIME::Parser to decode uuencoded parts, if it feels it can (and should) do it; - write warnings from MIME::Parser to the log (at log level 1 or higher), instead of discarding them. The package is available at the usual location at: http://www.ijs.si/software/amavisd/ where also the most up-to-date version of FAQ and certain other fresh documentation files live. --------------------------------------------------------------------------- April 24, 2002 amavisd-new-20020424 is primarily a maintenance release to summarize one week's worth of experience with amavisd-new-20020418 and to implement some good ideas from the amavis-user mailing list. It also brings one or two new features. It is available at: http://www.ijs.si/software/amavisd/ or more specifically, at: http://www.ijs.si/software/amavisd/amavisd-new-20020424.tar.gz Changes since amavisd-new-20020418: - removed Perl 'my' declaration from configurable variables which prevented them from being changed in amavisd.conf (thanks to Sebastian Hagedorn and Wouter de Jong for reporting) - introduced child timeouts to prevent bad amavis client from monopolizing a child forever (thanks to Sebastian Hagedorn for reporting the problem) - supported and documented (in README.postfix) configuration where multiple remote or local SMTP-in/SMTP-out MTAs (e.g. Postfix) can use the same amavisd server, by making it deliver checked mail back to the same IP address it came from (see variable $relayhost_is_client in amavisd.conf) (thanks to Wouter de Jong for the splendid idea which fitted naturally into the overall scheme) - in certain log messages include the SMTP-in and SMTP-out MTA IP address - new access list checking for IP addresses - used to limit SMTP access to authorized MTAs only: @inet_acl; access control is now enabled by default - slightly more sophisticated hash-based access list lookups, modeled after Postfix map lookups. The sequence now goes through the following steps: - hash lookup for user+foo@do.ma.in - hash lookup for user@do.ma.in (only if $recipient_delimiter is nonempty) - hash lookup for do.ma.in - hash lookup for .do.ma.in - hash lookup for .ma.in - hash lookup for .in - hash lookup user+foo@ - hash lookup user@ (only if $recipient_delimiter is nonempty) - updated README.lookups - cleaner quarantine code; new variable $mailfrom_quarantine allows to choose either the original envelope sender, or admin-specified fixed sender address; include a special per-user quarantine example (look for 'trouble-user-quarantine' in the amavisd) - fixed problem with localized system error messages - use numeric errno instead of strings in Sophie and Trophie clients (thanks to Igor D'Astolfo for reporting the problem with Italianized version of Linux and for a good suggestion) - successfully tested with Exim (thanks to Igor D'Astolfo for testing and for reporting the <> sender (but not recipient!) problem) - wrong variable used in the spam section ($final_virus_destiny instead of $final_spam_destiny) (thanks to Wayne Smith for reporting and for testing Vipul's Razor 'plugin' patch) - changed pattern match in the ./av/oav to support new version of OpenAntiVirus ScannerDaemon (thanks to Rainer Link) - changed 'configure' to make it recognize Sophie 1.33rc1 (thanks to Igor D'Astolfo and Lars Hecking) - include acinclude.m4/acx_pthread.m4 macro (thanks to Rainer Link) - updated README.postfix: includes instructions on how to avoid body_checks and header_checks for reinserted mail (contributed by Wayne Smith, works nicely, requires Postfix version 1.1.7-20020331 or later) - clarified comments in amavisd.conf - optionally keep existing X-Virus-Scanned: header lines, or remove them before adding our own header line - see $remove_existing_x_scanned_headers (requested by Darryl Harvey) - avoid historical (misleading) parameter name $localhost_ip; use $relayhost and $relayhost_port instead, but take old variables into account for backwards compatibility with existing amavisd.conf files - Here is an overall picture (sequence of events) of how pieces fit together: bypass_checks? ==> PASS no viruses? ==> PASS log virus if $log_templ is nonempty quarantine if $virus_quarantine_to is nonempty notify admin if $warnadmin notify sender if $warnsender notify recips if $warnrecip final_destiny==pass? ==> PASS virus_lovers? ==> PASS DISCARD or REJECT (depending on final_*_destiny) --------------------------------------------------------------------------- April 18, 2002 This it to announce the second release of amavisd-new-20020418, available at: http://www.ijs.si/software/amavisd-new-20020418.tar.gz It is a version of amavisd (a daemonized AMaViS, which is an interface between MTA and virus scanners), based on amavisd CVS from today (20020418) (same configuration, amavis clients in C, instructions, AV client code), while also being a successor of the initial release of amavisd-new-20020329, which is a performance-enhanced pre-forked Net::Server -based amavisd with SMTP-in/SMTP-out capability, written in Perl. (I tried to make this release based on amavisd-snapshot-20020300, but failed, as that version still has $errval semantics bug (the bitwise-operations problem was discussed on the AMaViS-user list some time ago). This was fixed in the CVS version and most AV clients were changed then. As the CVS version contains AV client code for new AV scanners, this makes it incompatible with config stuff from amavisd-snapshot-20020300. If you have problems with ./configure or make, these problems would be common to both versions. Also the unpackers and decompressors code is mostly the same for all recent amavisd versions, so any problems in this arena (like DoS mishandling) are most likely to be common for both the official amavisd and the amavisd-new version. Compared to the first release of amavisd-new-20020329, the second release brings further significant performance improvements especially in the SMTP-in/SMTP-out configuration, e.g. with Postfix, but also brings some interesting new features and new configuration possibilities. No important bug fixes were needed, so upgrade at your leisure if you are running my initial version. The Postfix users would perhaps want to rush a bit though, to put new performance improvements into use. The summary of changes since the initial version: - significant SMTP-in speedups (25% with fast AV scanner), file reuse - pass reject reason to MTA on the input side - more informative MTA log entries in the SMTP-in/SMTP-out setup - amavis internal id (am_id) in log entries and passed to MTA in SMTP response - ISP features: specify subgroups of users who want to receive viruses - address extensions: e.g. user@domain -> user+virus@domain if virus detected - can specify final_virus_destiny: reject, discard, pass - quarantine new options: save to individual file, save to mailbox, pass to MTA - new headers in quarantined viruses preserve envelope addresses and quarantine id (similar to the suggestion from Furio Ercolessi - see code) - detailed timing breakdown report for each passed message - anti-spam hooks and examples, example patch to integrate Vipul's Razor client - body cache now always enabled - heavy speedup for mailing list bursts - Sophie 1.33-ready - rewritten Trophie client, based on new Sophie client code - rewritten README.postfix, describing new setup possibilities - new file README.lookups (to be used with virus_lovers and bypass_checks) - new file README.customize (same as in the initial version) - new file README.performance (unfinished) - new amavisd.conf options, documented there - code heavily commented, cleaned, generalized again - does not accuse innocent users of sending viruses if we are suspicious of sender address (see FAQ below) (same as in the initial version) CAVEATS: - no test mode; - only Postfix and sendmail milter are fully supported and tested; to integrate with other mailers one would need some understanding of their operation to set up properly; contributions are welcome; (P.S. note: this is probably a non-issue, related to the file system problem: All amavisd versions seem to share one still unresolved problem, probably with amavis-milter.c client, its use of libmilter, or perhaps even in the libmilter code itself - see recent thread on the AMaViS-user list (subject: Leftover email.txt files with amavisd 'standard' also). The problem is most pronounced in burst of heavy traffic. ) FAQ: - Net::Server 0.82 triggers a Perl 5.005 bug (the problem is obvious: you get syntax errors). Either upgrade to Net::Server 0.83, or upgrade your Perl - 5.6.x should be ok. - if you intend to play with customized notifications, it is wise to remove the '-t' option from $sendmail_wrapper_args in amavisd.conf. That way you are free to screw up notification mail headers any way you want, and the message would still be delivered to the correct recipients. Removing '-t' is now the recommended setting, but both variants should work. If you use SMTP-out method for notifications (e.g. Postfix recommended and default setting), this does not affect you. - if you see virus notifications claiming the virus originator is or and sender notifications are not sent, this is not a bug, but a feature - see comments at the subroutine unmangle_sender(). The original idea comes from Furio Ercolessi: as some viruses tend to use forged or corrupted sender or 'From:' addresses, we try to determine the true virus sender, and if we can not do that, we avoid accusing innocent users of sending viruses. - if you kill or HUP amavisd, temporary directories may be left undeleted; this is normal and mails are not lost; - if amavisd does not restart after receiving HUP, a possible reason may be that amavisd can not be found in the path as set in the $ENV{PATH} variable (near the beginning of amavisd program). Another reason may be a syntax error if you changed the amavisd.conf file. Try to start it manually: $ su vscan -c amavisd If that does not make you wiser, set $DEBUG = "yes" and retry. This is also the recommended first-time start method. - after changing $inet_socket_bind in amavisd.conf, you must stop amavisd and start it anew. The HUP method causes amavisd to stumble over its feet. Below is my announcement notice for the initial version of amavisd-new, just slightly edited to remove some mistakes. --------------------------------------------------------------------------- Date: Sat, 30 Mar 2002 04:13:25 +0100 From: Mark Martinec Subject: [AMaViS-user] ANNOUNCE: new amavisd - leaner and meaner To: amavis-user@lists.sourceforge.net Message-id: <01KFYI1DJ2O200AMKT@CATHY.IJS.SI> I would like to announce an updated version of amavisd, based on recent CVS code of amavisd (which is not far from the February amavisd snapshot). It is a result of my three weeks work on the code, caused by our needs for: - better reliability; - higher throughput (less overhead); - versatility (e.g. separating amavis and MTA hosts, load sharing), Available at: http://www.ijs.si/software/amavisd-new-20020329.tar.gz This is now finally a version which I can recommend to friends :) TODO: better DoS handling in unpackers. Main features - in brief: - pre-forked reusable children - saving on process creations; - persistent connections to certain AV scanners, e.g. Sophie, saving on forks; - both SMTP and pipe (sendmail wrapper) interfaces independently available and configurable on all three sides: input, output, notifications; - sendmail Milter interface supported and tested; Postfix supported and thoroughly tested (Exim untested; classical sendmail untested, no qmail); - customizable notification messages; - compatibility with existing configurations; - cleanups, generalizations, speedups, fixes, better code documentation; - HUP signal causes restart with new configuration; - ISP feature: certain recipients may be allowed to receive viruses (with alert header line added (not with milter), notifications are still generated); - anti-spam hooks, caching-ready, Vipul's Razor interface in a form of a patch included - should make integration with SpamAssassin easier. Install: Unpack the tar over the checked-out CVS version of amavisd branch (or February snapshot). It overlays some files, the rest is unchanged. For milter interface make sure you use amavis-milter.c from CVS, as the snapshot version contains a bug which can cause message loss if amavisd dies. Details: - child and socket handling is now delegated to Perl module Net::Server, which gives us pre-forked children which are able to do more than one mail-check during their lifetime, saving on process creations and giving better response time; - Net::Server controls number of children, does signal handling, takes care of dead children, handles listening on multiple sockets (both Unix and TCP/UDP), delegating tasks and synchronizing 'accept's; - as a consequence, certain AV-scanner interfaces (most notably Sophie, and soon to follow Trophie) can keep persistent connections to the AV checking daemon, saving on AV scanner process creations (forks on accept), and socket setups/teardowns; - on the input side: both SMTP (TCP) and traditional amavis client protocol (Unix socket) are now accepted - even both at the same time (by default) to ease transition to SMTP interface; - new input-side SMTP interface (SMTP server) is easier to set up (no need to worry about file ownerships, UIDs and GIDs), and more versatile (e.g. SMTP responses carry more information that sysexits.h-based status codes, SMTP can talk to remote host); It is based on Perl native I/O and it is quite fast - no OO overhead, _not_ SMTP::Server-based (which I do not consider production quality); - a nice by-product of SMTP input interface is the extra information available in MTA logs, e.g. Postfix log: postfix/smtp[7656]: DA7B147FA6: to=, relay=localhost[127.0.0.1], delay=5, status=sent (250 Ok, discarded - VIRUS: EICAR-AV-Test) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - notifications messages can be delivered either via SMTP or as traditionally via pipe to sendmail wrapper / MTA pickup. This means that in the Postfix setup for example, one needs to worry only about one output interface (SMTP or pipe). This also saves unnecessary work of checking the just-generated notification messages for viruses; - the output pipe interface is now more careful with status checking and forking; - output SMTP interface error handling more compact; - sender notifications can be sent to more than one contact address (not used at the moment, but the mechanism is available); - sender address unmangling patch incorporated (but I'm willing to take it out if considered inappropriate for the base distribution); - see file ./amavis/mta/postfix_init for comments describing how to select SMTP- or pipe-based output interface (for re-injection and notifications); - customizable notification messages and log entry text (see README.customize); - clean notion of when mail addresses are in their quoted and when in unquoted form (RFC2821). All internal handling uses unquoted form, addresses get quoted as required by the output interface, and quoting gets stripped away as required by each input interface; - besides traditional choices where a mail can be forwarded/accepted, discarded, or temporary failure/retry indicated, there is now a fourth choice: REJECT. Depending on the MTA this requires a message rejection to be done by the input side MTA itself. This comes handy in cases when amavisd accepts a non-infected mail, but outgoing MTA does not want to take it back for final delivery, e.g. in case of some policy violation. Traditionally amavisd would indicate temporary failure on its input side, causing the message to be retried and re-scanned over and over again, without having a chance of ever being accepted; - all existing virus scanners are still there (one little change in each: replaced a call to do_virus with return 1); Sophie client rewritten to take advantage of persistent connections; - decoding sections are mostly unchanged. This area needs more work in the future; - includes exit status codes from sysexits.ph instead or having them hard-wired; - anti-spam code easier to integrate into amavisd due to some code rearrangements; a patch to integrate Vipul's Razor is included as an example, but it may be better to tie amavisd with SpamAssassin. Anti-spam code will NOT be integrated into amavisd, but the least we can do is making it easier for people to add their own code; - cache-ready (example in the included Razor patch). This works by calculating a message digest (hash, signature) of the message body and keeping it in storage for a short while, e.g. for the lifetime of a child process (10 consecutive requests by default). If another message with the same body content arrives in the near future we can skip a virus check. This comes handy where mailing list traffic is frequent, especially if we have to deal with poorly done mailing list managers or heavy spam traffic; To support this concept the functions of MIME decoding and unpacking of archives are now separated. - MIME parsing is now supplied with our own Filer subclass. This was necessary to avoid MIME parser complaints when it tried to reconstruct file name extensions from file names in strange character sets. Its work was completely unnecessary and harmful, since we want to supply our own file names and do not care for file name extensions. - as a consequence of our own Filer subclass, we now avoid the first (or the only) directory traversal (reading) in the first decoding pass; Not much, but every little bit counts. - when using sendmail wrapper it is no longer necessary to supply the sendmail -t option. If -t is not specified, sendmail wrapper receives addresses via command arguments (exec, no shell), which is more reliable than having to parse mail headers - which are now more error-prone due to user-customizable notifications. The -t option is still supported though, but not recommended. - do_unzip no longer complains with multi-line backtrace when it dies; just a single message is issued, like with other decoders; - HUP signal causes restart and re-reading of config file; - many new comments, code unifications, supplied some missing error checks, code generalizations; MTA support: - thoroughly tested with Postfix in all combinations of input, output, and notifications interfaces; in production use; - big thanks to Sebastian Hagedorn who helped to test the milter interface on his Solaris! - Exim and traditional sendmail interface untested, but should work without much work. - Qmail interface is still missing, like in current CVS amavisd. I believe the amavis client for qmail can be written as a few-dozen line Perl program. Anybody want to try? A quick cookbook on how to set up Postfix / amavisd interface using SMTP on both amavisd input and output side (including notifications). Amavisd by default now accepts both Unix socket and SMTP on the loopback interface, so the transition is easier: first install new amavisd, then at some other time change Postfix configuration (if desired) to: master.cf: # MTA -> amavisd smtp-amavisd unix - - n - 2 smtp # amavisd -> MTA localhost:10025 inet n - n - - smtpd -o content_filter= main.cf: # choose transport to amavisd content_filter = smtp-amavisd:localhost:10024 [see new file README.postfix for details] The amavisd can now easily be located on a different host than MTA, also Postfix load-balancing transport methods can be used (e.g. multiple MX records). Although amavisd now talks SMTP and incorporates some rudimentary defenses against malicious SMTP clients, do not expose its SMTP server directly to the world - always front-end it with MTA. By default it binds to the local interface only, but other access restrictions are also available. Happy amavising! Experiences and comments are most welcome.