News Group Posts

save log all mail messages

Before milters, the only choice for archiving sendmail mail messages was "-x", which (as noted below) created very large log files.

Newsgroups: comp.unix.sco.misc
From: bill@wjv.com (Bill Vermillion)
Subject: Re: How to save ALL inbound and outbound mail
Message-ID: <G13CzH.10Au@wjv.com> 
Date: Mon, 18 Sep 2000 16:40:29 GMT

In article <39C28972.6EC08990@aplawrence.com>,
Tony Lawrence  <tony@aplawrence.com> wrote:
>Fabio Giannotti wrote:

>> Hello all,

>> Running 5.0.5 with sendmail 8.8.8.

>> Does anyone know how to do sendmail.cf "magic" to allow the
>> server to keep a copy of all inbound and outbound mail?

>> (Client needs this for legal CYA reasons...)

>When you start the daemon, add "-X
>/var//log/monster_maill_log".  You'll need to stop sendmail
>fairly often to compress and rotate those logs; they will
>get monstrous.. I do this at several client sites probably
>for the same reasons your client wants to do it- we keep a
>few days on the server and the rest can be had from tape if
>ever needed..

Thanks for that pointer Tony.  So many programs - so many flags.

I just tried it and I saw why it gets 'monstrous'.  It logs the file
incoming and delivery too. That's all the handshake headers and 2
copies of the message. I just sent a 1 line email message
- plain text - 43 characters long.  The log file is 3512.
Monstrous is almost an understatement.

Looking at it I had a small thought.  Besides compressing and
rotating the logs, you could pre-process them by doing something as
grep "<<<" <logfile.name> > incoming.messages and 
grep ">>>" <logfile.name> > outgoing.messages, of course
using part of the date command to give you dated message files.
That would make things a trifle bit more manageable.

Thanks for expanding my knowledge base.

Bill

-- 
Bill Vermillion -   bv @ wjv . com

From - Wed Sep 20 17:35:41 2000
From: "Fabio Giannotti" <fabiog@venmar.com>
Newsgroups: comp.unix.sco.misc
Subject: Re: How to save ALL inbound and outbound mail
X-MSMail-Priority: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300
Message-ID: <Bi3y5.429$b64.65372@news1.primary.net> 
Date: Wed, 20 Sep 2000 08:51:50 -0500


Bill Vermillion <bill@wjv.com> wrote in message news:G1560D.16Ht@wjv.com...
> In article <39C78096.C9A14F7A@aplawrence.com>,
> Tony Lawrence  <tony@aplawrence.com> wrote:
> >Fabio Giannotti wrote:
>
> >> Thanks to all who responded!
>
> >> I am trying to go with Tony's solution of -X logfile, but am
> >> having a little problem.
>
> >> On a test system with low volume, it works great. However, on the
> >> real system with fairly heavy volume, it does not log anything,
> >> and I keep getting the following error in syslog:
>
> >> NOQUEUE: SYSERR(root) opendaemon socket: server SMTP socket
> >> wedged: exiting
>
>
> >I've only used this on Linux systems with high volume..
>
> I have noticed while testing the -X you suggested, that mail
> delivery delay on a local station increases.   Besides copying the
> message three times there is the overhead of inserting all the
> tracing information, process ID, etc., before each line in the
> log file.
>
> I guess what is needed is something like a simple tee function so
> the message is just copied to two places instead of processing the
> message on a line by line basis.  Fabio didn't say just how large
> his mail volume was - just that it was large.
>
> I wonder what something such as sar would say on disk useage and
> CPU time during this.  I suspect the mail handler just got too busy
> logging.  The logfile is 7 times larger than the mail message in
> the user mailbox.
>
>
> --
> Bill Vermillion -   bv @ wjv . com

Update to all who helped with this problem.

For safety (read: being afraid of new releases) reasons, I will probably go
with Roberto's script (Thanks Roberto!), but I also found out the following:

It turns out that Sendmail versions 8.10 and higher provide a hook that
allow you to write macros to do all sorts of things to mail as it passes
through the system.

The sample macro they give you, of course, is to save a copy of all email
going through the system!

So, my question NOW is:  Has anyone compiled Sendmail 8.10 or better on
OpenServer 5.x, and if so, can I get a copy of it and how well does in work?



From - Wed Sep 20 17:35:58 2000
From: "Fabio Giannotti" <fabiog@venmar.com>
Newsgroups: comp.unix.sco.misc
Subject: Re: How to save ALL inbound and outbound mail
X-MSMail-Priority: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2314.1300
Message-ID: <N27y5.1$R04.316@news1.primary.net> 
Date: Wed, 20 Sep 2000 13:08:03 -0500


Bill Vermillion <bill@wjv.com> wrote in message news:G16xy5.1Dzz@wjv.com...
> In article <Bi3y5.429$b64.65372@news1.primary.net>,
> Fabio Giannotti <fabiog@venmar.com> wrote:
> >
> >Bill Vermillion <bill@wjv.com> wrote in message
news:G1560D.16Ht@wjv.com...
> >> In article <39C78096.C9A14F7A@aplawrence.com>,
> >> Tony Lawrence  <tony@aplawrence.com> wrote:
> >> >Fabio Giannotti wrote:
>
>
> >> >> I am trying to go with Tony's solution of -X logfile, but am
> >> >> having a little problem.
>
> >> >> On a test system with low volume, it works great. However, on the
> >> >> real system with fairly heavy volume, it does not log anything,
> >> >> and I keep getting the following error in syslog:
>
> >> >> NOQUEUE: SYSERR(root) opendaemon socket: server SMTP socket
> >> >> wedged: exiting
>
> >> >I've only used this on Linux systems with high volume..
>
> >> I have noticed while testing the -X you suggested, that mail
> >> delivery delay on a local station increases.   Besides copying the
> >> message three times there is the overhead of inserting all the
> >> tracing information, process ID, etc., before each line in the
> >> log file.
>
> >> I guess what is needed is something like a simple tee function so
> >> the message is just copied to two places instead of processing the
> >> message on a line by line basis.  Fabio didn't say just how large
> >> his mail volume was - just that it was large.
>
> >> I wonder what something such as sar would say on disk useage and
> >> CPU time during this.  I suspect the mail handler just got too busy
> >> logging.  The logfile is 7 times larger than the mail message in
> >> the user mailbox.
>
>
> >Update to all who helped with this problem.
>
> >For safety (read: being afraid of new releases) reasons, I will
> >probably go with Roberto's script (Thanks Roberto!), but I also
> >found out the following:
>
> >It turns out that Sendmail versions 8.10 and higher provide a hook
> >that allow you to write macros to do all sorts of things to mail as
> >it passes through the system.
>
> >The sample macro they give you, of course, is to save a copy of all
> >email going through the system!
>
> Care to mention the title of this macro - so I don't have to hunt
> everywhere for it :-).  I know, I know.  RTFM - but it is SO BIG.
> :-)

Sorry 'bout that.

The command/macro/whatever is called "milter" and it has a related
"LIBMILTER" associated with it.  Keep in mind I have no idea what I'm
talking about. ;-)
Fabio


From - Tue Oct  3 05:03:30 2000
From: "Otto Rodusek (SG)" <aplcom@cyberway.com.sg>
Newsgroups: comp.unix.sco.misc
Subject: Re: tracking outgoing mail (Got soln!)
Date: Tue, 3 Oct 2000 07:24:27 +0800
X-MSMail-Priority: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200
Message-ID: <39d9199b@news.starhub.net.sg> 

Yo,

1.    I've been following this thread for some time and finally took the
bull by the horns and wrote a buncha scripts that get the job done. The end
result is that I have a copy of ALL outgoing/incoming mail in a single
repository this is fully mail "readable" in full mail format.

2.    If any one is interested, drop me a line a aplcom@cyberway.com.sg and
I'll email you whichever of the scripts you need. Below is a description of
my problem, the environment I use and the solution that I came up with.

Enjoy. Rgds. Otto.

=======================================================

I'm contracted to a company that doesn't want to spend a whole load of cash
on maintaining a complex email system nor the cash to have a permenant ISP
link up. So I had to come up with a method to have full scale email but at a
very CHEAP price.

a/    We are using SCO 5.0.5, fetchmail 5.2.7, sendmail 8.8.8  (all as
originally bought/received from SCO)

b/    Basic hardware setup with a K7-650, 16Gb ultra scsi, 24 GB dat backup
(Microlite), 32 port Specialix RIO, 48 port ethernet 10/100 .

c/    We have a hosted web and email site. I have instructed our hosted web
site to capture ALL incoming mail into a single account (rootbag).

d/    On system boot, sendmail is started from /etc/rc2.d/P86sendmail with
the following params (-bd -X /usr/adm/log/mailogd) - I have found that this
setting will deliver ALL local mail but queue (/usr/spool/mgueue) all
outgoing mail not local to my server.

e/    Every 30 minutes cron wakes up a script called pppgo:

        PPPGO - shell script that  establishes and verifies ppp link via
modem to our ISP
                         performs fetchmail to retrieve the mailbag into
account rootbag
                         makes/stores a copy of the mailbag into account
(mailin)
                         runs a shell script called maildist to distribute
all mail in rootbag to all valid users
                         runs sendmail (-v -q -X /usr/adm/log/maillog)
                         runs a shell script to store ALL outgoing mail into
account (mailout)
                         closes the ppp link and goes to sleep
                         inits all the maillog files!!! (this way kept to a
VERY reasonable size - a few K)

        MAILDIST    - Shell script that reads all the mail in rootbag and
distributes to all valid users as defined in /etc/passwd. If not found in
/etc/passwd gets mailed to our accounts executive who then decides who the
mail is really for.

        MAILOUT    - Shell script that read the sendmail logs and creates a
file similar to rootbag but contains properly formatted email in proper SCO
mail format and pops it all into an account called mailout. Mailout can then
be read by any mail reader - we use outlook express.

rgds. otto.

Comments?

More Articles by Tony Lawrence

Jump to Comments

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them. I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.

Specific links that take you to pages that allow you to purchase the item I reviewed are very likely to pay me a commission. Many of the books I review were given to me by the publishers specifically for the purpose of writing a review. These gifts and referral fees do not affect my opinions; I often give bad reviews anyway.

We use Google third-party advertising companies to serve ads when you visit our website. These companies may use information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide advertisements about goods and services of interest to you. If you would like more information about this practice and to know your choices about not having this information used by these companies, click here.