[Users] Downloads for offline use need to be multi-threaded for speed

[Gene Heskett]

On Sunday 07 September 2014 22:01:55 blind Pete did opine
And Gene did reply:
> On Sun, 7 Sep 2014 10:59:25 -0400
> 
> Gene Heskett <gheskett at wdtv.com> wrote:
> > On Sunday 07 September 2014 09:32:18 blind Pete did opine
> > 
> > And Gene did reply:
> > > On Sat, 6 Sep 2014 20:41:35 +0530
> > > 
> > > Rajib Bandopadhyay <bkpsusmitaa at gmail.com> wrote:
> > > > Dear Friends,
> > > > 
> > > > The pre-processing filters make
> > > > claws-mail terribly slow to download
> > > > posts from newsgroups.
> > > > 
> > > > The speed could be sufficiently improved
> > > > if the downloads are made
> > > > multi-threaded.
> > > > 
> > > > Is it possible to improve upon the
> > > > download speed?
> > > > 
> > > > Could this be implemented please?
> > > > 
> > > > Regards
> > > > Rajib Bandopadhyay
> > > 
> > > Consider installing a news spool.  I use leafnode, but there are
> > > others.  The news spooler periodically downloads news posts in
> > > the background, then your news reader only has to fetch items
> > > from the hard drive.
> > > 
> > > One day I have to get around to configuring dovecot to do the
> > > same job for mail.
> > 
> > That would probably, in the long view, be a mistake.  I am still
> > doing everything by pop3 here, and while kmail CAN do that easily
> > enough, its a job its ill suited for because its single threaded.
> 
> Kmail?
> 
> > So I have, for nearly a decade now, offloaded that mail fetching duty
> > to fetchmail, which hands it off to procmail for filtering and
> > delivery into a /var/spool/mail/name mailfile.
> 
> Books have been written about configuring mail servers.  I find the
> subject a bit daunting.  It would be embarrassing to find that my
> computer was running an open mail relay.  So do it properly or don't
> do it at all, is my position.

I never said anything about a server.  Fetchmail sends mailfilter out to 
check my 3 ISP accounts, one of which is not an actual ISP, but is my 
former employers own mail server, a qmail install that is now 15 years 
old.  My old employee account is a lifetime account IOW.  The other two 
are gmail hosted, and while I nuke a googlegroups message on reception, I 
do tolerate gmail as its not nearly so screwed up

When mailfilter is finished, and it checks for known spam and deletes a 
few of them in the process, but I have to compose the killfile list 
myself.  Then whats left, fetchmail downloads and hands it off to 
procmail, which runs it thru clamav, and passing that, then runs it thru 
spamassassin, which will reroute some of it to the spam folder so I can 
check to see if it really was spam.  If something gets nailed, I can then 
move it to the ham folder, and a few minutes after midnight a session of 
sa-learn --ham is run on the ham folder, and another session of sa-learn 
--spam is run on the spam folder, activities intended to train 
spamassassin.  All cron jobs that once setup can be forgotten as they Just 
Work(TM). 

Whats left is then written to /var/spool/mail/gene, and when that file is 
closed, inotifywait triggers, a dbus message is sent to kmail telling it 
to go get the local mail, which it can do in 5 to 200 milliseconds.  There 
is a bit more to it, but thats the invisible blow by blow.

Yes, I've been fine tuning on bash scripts for about 15 years now, its 
actually a quite capable script interpreter.  Use it, and cron to make 
your online life 90% easier.

My biggest problem with kmail is that my email corpus is some north of 
20Gb, and it doesn't do its housekeeping often enough, so when about 1000 
read emails have been processed, it starts taking way too long for the 
plus key to bring up the next message.  But I've found that stopping it, 
which is where it does some of that housekeeping, and immediately 
restarting it brings back the nice snappy next message function.

My second biggest problem with kmail is that the various account settings 
each sent message references, all have sticky checkboxes, but they do not 
work about once a day, so before I click send, I have to read the header 
and make sure its going out through the correct account, the one I used to 
subscribe to the list with.

Other than those 2 items, I am a happy kmail user since about 1999.
 
> You look like you know what you are doing, but I have not come to
> terms with rather a lot of details.
> 
> Why don't you like dovecot?

Dovecot is not an email agent, it is an imap server.  For starters, the 
problems I see going by on that list are the teething problems of a new 
protocol.  But neither Dovecot nor imap are newbies.  It also adds 
complexity to the email system, IMO way more than my "bandaids" that fix a 
specific problem are.  By now it should be fairly mature, but its not 
acting that way to me.

For 2nd starters, neither claws nor Dovecot has any facilities to allow it 
to import the mixture of maildir and mailfile kmail is using.  And yes, I 
have asked, on both this list and the Dovecot list at least 3 times about 
that.  A new file format, mndir? or at least a new name for something that 
from what I read here, acts much like a maildir.  But I haven't found a 
description so I could understand the difference.  Or even why the 
difference exists.  There must be an advantage to it, but even that 
doesn't seem to have been discussed since I subscribed to the list about 1 
year back in anticipation of having to upgrade to at least ubu 14.04.2. It 
seems to be treated as if everybody should intimately know already.

> > So regardless of your favorite email agent, off loading the online
> > mail fetching activities to something designed to be a background
> > task will be a genuine enhancement to your enjoyment because you'll
> > never have to wait 30+ seconds for it to process and deliver your
> > incoming mail in the middle of typing an answer to a previously
> > received email.
> > 
> > Cheers, Gene Heskett
> 
> There are long undesirable pauses during network activity.

With my way of doing it, those pauses are perhaps 200ms long worst case 
now.  It  doesn't kmail take very long to fetch a new message from the 
local mail spool.

But I am not trying to badmouth claws too much, the nightmares with the 
newer versions of kmail continue apace on the ubu lists.  And some of them 
are IMO not excusable, one fellow lost his inbox, with messages several 
years old the other day.  That's downright scary.  I suspect it was a 
loose nut behind the wheel that caused it, but still...  And of course the 
guy has no clue what a backup is.  Amanda saves this system nightly.

I can have that inbox recovered to the status it was in at about 1:30 AM 
in perhaps 15 to 20 minutes.  I do backups as virtual tapes, but on a big 
disk so a recovery doesn't involve a long session of reading a tape, 
forced because the tape is sequential access, but the disk is truly 
random.

This has turned into the weekly fishwrap, the only thing missing is the 
obit pages. ;-)

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>
US V Castleman, SCOTUS, Mar 2014 is grounds for Impeaching SCOTUS