[Users] Downloads for offline use need to be multi-threaded for speed
gheskett at wdtv.com
Mon Sep 8 05:17:18 CEST 2014
On Sunday 07 September 2014 22:01:55 blind Pete did opine
And Gene did reply:
> On Sun, 7 Sep 2014 10:59:25 -0400
> Gene Heskett <gheskett at wdtv.com> wrote:
> > On Sunday 07 September 2014 09:32:18 blind Pete did opine
> > And Gene did reply:
> > > On Sat, 6 Sep 2014 20:41:35 +0530
> > >
> > > Rajib Bandopadhyay <bkpsusmitaa at gmail.com> wrote:
> > > > Dear Friends,
> > > >
> > > > The pre-processing filters make
> > > > claws-mail terribly slow to download
> > > > posts from newsgroups.
> > > >
> > > > The speed could be sufficiently improved
> > > > if the downloads are made
> > > > multi-threaded.
> > > >
> > > > Is it possible to improve upon the
> > > > download speed?
> > > >
> > > > Could this be implemented please?
> > > >
> > > > Regards
> > > > Rajib Bandopadhyay
> > >
> > > Consider installing a news spool. I use leafnode, but there are
> > > others. The news spooler periodically downloads news posts in
> > > the background, then your news reader only has to fetch items
> > > from the hard drive.
> > >
> > > One day I have to get around to configuring dovecot to do the
> > > same job for mail.
> > That would probably, in the long view, be a mistake. I am still
> > doing everything by pop3 here, and while kmail CAN do that easily
> > enough, its a job its ill suited for because its single threaded.
> > So I have, for nearly a decade now, offloaded that mail fetching duty
> > to fetchmail, which hands it off to procmail for filtering and
> > delivery into a /var/spool/mail/name mailfile.
> Books have been written about configuring mail servers. I find the
> subject a bit daunting. It would be embarrassing to find that my
> computer was running an open mail relay. So do it properly or don't
> do it at all, is my position.
I never said anything about a server. Fetchmail sends mailfilter out to
check my 3 ISP accounts, one of which is not an actual ISP, but is my
former employers own mail server, a qmail install that is now 15 years
old. My old employee account is a lifetime account IOW. The other two
are gmail hosted, and while I nuke a googlegroups message on reception, I
do tolerate gmail as its not nearly so screwed up
When mailfilter is finished, and it checks for known spam and deletes a
few of them in the process, but I have to compose the killfile list
myself. Then whats left, fetchmail downloads and hands it off to
procmail, which runs it thru clamav, and passing that, then runs it thru
spamassassin, which will reroute some of it to the spam folder so I can
check to see if it really was spam. If something gets nailed, I can then
move it to the ham folder, and a few minutes after midnight a session of
sa-learn --ham is run on the ham folder, and another session of sa-learn
--spam is run on the spam folder, activities intended to train
spamassassin. All cron jobs that once setup can be forgotten as they Just
Whats left is then written to /var/spool/mail/gene, and when that file is
closed, inotifywait triggers, a dbus message is sent to kmail telling it
to go get the local mail, which it can do in 5 to 200 milliseconds. There
is a bit more to it, but thats the invisible blow by blow.
Yes, I've been fine tuning on bash scripts for about 15 years now, its
actually a quite capable script interpreter. Use it, and cron to make
your online life 90% easier.
My biggest problem with kmail is that my email corpus is some north of
20Gb, and it doesn't do its housekeeping often enough, so when about 1000
read emails have been processed, it starts taking way too long for the
plus key to bring up the next message. But I've found that stopping it,
which is where it does some of that housekeeping, and immediately
restarting it brings back the nice snappy next message function.
My second biggest problem with kmail is that the various account settings
each sent message references, all have sticky checkboxes, but they do not
work about once a day, so before I click send, I have to read the header
and make sure its going out through the correct account, the one I used to
subscribe to the list with.
Other than those 2 items, I am a happy kmail user since about 1999.
> You look like you know what you are doing, but I have not come to
> terms with rather a lot of details.
> Why don't you like dovecot?
Dovecot is not an email agent, it is an imap server. For starters, the
problems I see going by on that list are the teething problems of a new
protocol. But neither Dovecot nor imap are newbies. It also adds
complexity to the email system, IMO way more than my "bandaids" that fix a
specific problem are. By now it should be fairly mature, but its not
acting that way to me.
For 2nd starters, neither claws nor Dovecot has any facilities to allow it
to import the mixture of maildir and mailfile kmail is using. And yes, I
have asked, on both this list and the Dovecot list at least 3 times about
that. A new file format, mndir? or at least a new name for something that
from what I read here, acts much like a maildir. But I haven't found a
description so I could understand the difference. Or even why the
difference exists. There must be an advantage to it, but even that
doesn't seem to have been discussed since I subscribed to the list about 1
year back in anticipation of having to upgrade to at least ubu 14.04.2. It
seems to be treated as if everybody should intimately know already.
> > So regardless of your favorite email agent, off loading the online
> > mail fetching activities to something designed to be a background
> > task will be a genuine enhancement to your enjoyment because you'll
> > never have to wait 30+ seconds for it to process and deliver your
> > incoming mail in the middle of typing an answer to a previously
> > received email.
> > Cheers, Gene Heskett
> There are long undesirable pauses during network activity.
With my way of doing it, those pauses are perhaps 200ms long worst case
now. It doesn't kmail take very long to fetch a new message from the
local mail spool.
But I am not trying to badmouth claws too much, the nightmares with the
newer versions of kmail continue apace on the ubu lists. And some of them
are IMO not excusable, one fellow lost his inbox, with messages several
years old the other day. That's downright scary. I suspect it was a
loose nut behind the wheel that caused it, but still... And of course the
guy has no clue what a backup is. Amanda saves this system nightly.
I can have that inbox recovered to the status it was in at about 1:30 AM
in perhaps 15 to 20 minutes. I do backups as virtual tapes, but on a big
disk so a recovery doesn't involve a long session of reading a tape,
forced because the tape is sequential access, but the disk is truly
This has turned into the weekly fishwrap, the only thing missing is the
obit pages. ;-)
Cheers, Gene Heskett
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>
US V Castleman, SCOTUS, Mar 2014 is grounds for Impeaching SCOTUS
More information about the Users