grep on RFC-822 headers and stuff

[Kragen Sitaker]

I want something that's partway between a database and a text file.  I
want to be able to arbitrarily annotate entries, add and delete fields
at will from individual records, and also do databasish stuff with it.
Like askSam.  Also, being able to have multiple instances of a field
can be awfully helpful sometimes.

A few ideas:
- RFC-822 is a great format
- Entering things in RFC-822 format could be greatly eased by 
  'move to next field' and 'move to previous field' keys, along with
  a key to insert empty fields from some schema, and automatic
  indentation.  And working paragraph filling!  Emacs paragraph fill works
  very badly when you have hanging-indent paragraphs, especially with
  no blank lines between them
- not everything need be in a field
- fields can be defined more liberally than in RFC-822; an initial nonempty
  string of nonwhitespace on a line can start a new field which continues
  until the beginning of the next field; indentation is useful but not 
  required.  Indentation should be stripped.
- fields could be nested with indentation.
- separating RFC-822 records in a file is somewhat problematic, but a : 
  on a line by itself could be used.
- hypertext would be nice --- ideally you could put a query inside some kind of
  special characters to take you to the set of items that satisfied it.  Like
  [title: A Day To Mourn].  A little syntax-highlighting to let you know it was
  a link, whether or not it led to anything, and a status-line cue
  when you moved into it or clicked on it to let you know how to follow it 
  would be good.
- being able to have multiple fields on the same line would be really good.
  Perhaps there could be an alternate newline character or character
  sequence (for purposes of field delimiting): one of ; | ) } ] ! $
  ... -- :: could work.  Maybe just lots of whitespace?

  from-page: 10  ;to-page: 12   ;has-pictures: no
  from-page: 10  |to-page: 12   |has-pictures: no
  from-page: 10  )to-page: 12   )has-pictures: no
  from-page: 10  }to-page: 12   }has-pictures: no
  from-page: 10  ]to-page: 12   ]has-pictures: no
  from-page: 10  !to-page: 12   !has-pictures: no
  from-page: 10  $to-page: 12   $has-pictures: no
  from-page: 10  ...to-page: 12   ...has-pictures: no
  from-page: 10  --to-page: 12   --has-pictures: no
  from-page: 10  ::to-page: 12   ::has-pictures: no
  from-page: 10    to-page: 12     has-pictures: no

  Of these, I like ..., ;, and $ best.
  This conflicts somewhat with nested fields.  Perhaps this could be
  resolved by saying that all fields that begin on the same line must
  be siblings.

I should look at Hyperbole.  Maybe it's already something like this.
How about the forms package in xemacs?  Widget?

On queries: you need both select (pick particular records) and project
(pick particular fields of records).  There should probably be default
projections for queries, which should be specified in the file itself.
I'd like select to be pretty query-by-example --- ideally, the select
queries would be ordinary-looking records --- and I'd like select
results to display incrementally.  I'd also like brief select results
to be hyperlinks to the full records, and they should be either
ordinary-looking records or queries.

My present design: 

A file is a bag of records.  Each record is both a string and a
(possibly-empty) bag of fields.  Fields are associations between
fieldnames and records --- that is, the value of a field is a record
in its own right.

Since records are possibly-empty bags of fields, the expression
somerec.somefieldname can refer to zero, one, or more values in
somerec.

Records that aren't empty contain at least one line that doesn't begin
with whitespace; to meet this criterion, whitespace is trimmed from
the beginnings of all lines until it's met.  (That's not quite right,
because "x: a\n   b\n   c" should contain "a\nb\nc", with no spaces.)

Records at the top level in the file might have an additional
invisible field: recno.

Files can contain queries (normally with update) that are run when the
file is loaded, and again upon command.  This is useful to maintain
calculated, or virtual, fields.  I think this is simpler than having
another layer of indirection between the data and the display.  People
should be warned when they're editing a calculated field by hand.

There can be a field description record that gives textual
descriptions of fields, possibly even validity constraints.  (I don't
believe in enforcing validity constraints, but I do believe in making
the field a different color and giving the user a message.)


I guess I should write down why I want this.  


One application is my grocery price list.  I've been keeping track of
how much groceries cost when I buy them.  I'd like to be able to get a
list of the lowest prices I've seen on groceries, get a list of the
lowest prices I've seen on just staple groceries, the most recent
prices I've seen on everything, the lowest prices I've seen in the
last six months, and flexibly include or exclude things like unit
cost, date bought, etc.  And, given such a list, I want to be able to
edit things in it and go back to the full original record it came from.

Another application is keeping notes on research papers in computer
science.  I'd like to keep items such as URL, author, title, date,
subject areas, and location on disk, as well as the full text of the
paper, in some kind of searchable database.  I'd like to be able to
extract particular items from it in BibTeX format, or just plain text
format.  And I'd like to be able to view the paper, too.

A third application is keeping track of my contacts; I'd like to know
when I've last contacted or heard from someone and how often I
normally contact them, so I can bring up a sorted list of the next ten
people I should touch base with.

A fourth application is keeping track of notes like these, making it
possible to remind me to work on things or to touch base with someone
I'm waiting for.

A fifth application is keeping track of restaurants to which I've been
or would like to go.

A sixth is general to-do list items, for which I want simple summary
views, the ability to easily mark things as done, arbitrary
annotation, and archival --- the ability to see things that are not
yet done, that weren't done at some point in the past, etc., and
full-text search for old items.

-- 
<kragen@pobox.com>       Kragen Sitaker     <http://www.pobox.com/~kragen/>
Perilous to all of us are the devices of an art deeper than we possess
ourselves.
       -- Gandalf the White [J.R.R. Tolkien, "The Two Towers", Bk 3, Ch. XI]