Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Thu, 2 Feb 2012 07:42:12 +0400
From: Solar Designer <solar@...nwall.com>
To: owl-dev@...ts.openwall.com
Subject: Re: blists status?

Hi Lauri,

blists is not (yet) part of Owl, so this is slightly off-topic for this
mailing list.  That said, I don't mind discussing it in here briefly
before we decide on next steps.  And we might in fact include blists
into Owl after we get a web server into Owl base tree (there's almost no
point in having blists without a web server).

It is impressive how you analyzed blists before bringing this topic up.
Are you using blists somewhere?

On Tue, Jan 31, 2012 at 08:30:10AM +0200, Lauri Kokkonen wrote:
> I have been considering to work on blists and want to ask if this is
> something you would like to see worked on during GSoC.

Maybe - if we're in GSoC again and if we have an excellent match -
although blists for this year's GSoC was not specifically planned
(rather, there are blists tasks assigned to abc - although as you can
see he's been inactive at blists lately, so re-assigning is possible).

> Looking at the source there seem to be potential portability issues. First,
> since the index is read directly from file to structs there arises
> endianness problems when carrying the index file from LE arch to BE, for
> instance. Second, structs might be padded differently on different archs.

Definitely.  In its current form, bindex and bit are supposed to be from
the same build of blists and running on the same system.

> Although maybe this is not serious since the index can just be generated
> from the beginning, and fixing those issues will probably just make bindex
> slower.

Yes, although I am more concerned about source code size and complexity
increase for a relatively rare use case (e.g., with bit and bindex on
different machines accessing shared storage).

> The HTML output stuff has some duplicate code at least for reading the aday
> index and printing a list of messages for month and day indices, and might
> benefit from a cleanup.

Perhaps.

> For new features I would personally like to see threaded view.

Yes, it's one of the desirable features.

> This should be possible to add without changes to the index format.

Currently, the threads are "flattened" - each message only has one
thread-prev and one thread-next message - that's in the index file.
(Yes, this means that thread-prev is not necessarily the message that
the current message is a direct reply to.  It may also be a message
posted to the same thread after the replied-to message was posted.)

So for a flat threaded view, everything is easy (and this may be an
initial development milestone).

For a tree threaded view, we may either scan the flattened thread and
match instances of msgid_hash vs. irt_hash (in bit) or enhance the index
format and record more-direct pointers up and down in the tree in the
index.  I currently think that the former approach is preferable, but
this may need further consideration.

> Another often seen
> feature is author view but I am not so sure about it; do you want to
> build a rich set of features or go with a compact piece of software?

We need to strike a balance.  Besides code size, also important are code
complexity and potential security risks.  Merely adding a new display
mode is likely low-risk, whereas e.g. adding a search feature is higher
risk (extra user input).  That said, author view was not requested so
far, so it is not on the agenda, whereas search was requested, so it is.
Also, maybe having "advanced search" would eliminate the need for author
view (if there's such need at all).

> A search feature would be wonderful. Probably some kind of search index that
> links the search term space to individual messages has to be built for
> this. There are many different options on how to do this, and some problems

Right.  abc looked into using Xapian for this - in fact, he experimented
with it briefly in a modified blists tree.  He would probably proceed to
implement it for real by now, but we agreed on other priorities
initially - such as index pages with Subjects and Froms (already in
blists 1.0), "Recent messages" lists (in CVS), support for different
character encodings (in CVS for headers, not yet implemented for bodies).

> like the order of mbox index entries possibly changing (if I have understood
> correctly),

Why would it change?  Are you possibly referring to the qsort() call?
If so, that is something abc ran into too and we discussed ways around
it - but I forgot the details (may need to dig up chat logs).

> so that it probably needs a few candidate implementations and
> serious testing.

Definitely.  And frankly, I am concerned about running a version of bit
linked against Xapian or whatever and passing untrusted user input from
the web into that library...  I am not sure how to deal with this best.

> Do you have other TODO items for blists?

Yes.  I have a cryptic TODO list originally intended for myself only
(before abc got involved in blists development).  To name a few items
(not yet mentioned above): support for downloads of attached files, RSS
or Atom feed, highlight search keywords based on Referer (such as from
Google search), and many smaller enhancements.

Thanks,

Alexander

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.