Hence "memory" - because that's how people use it :)
I agree. Although, it's more than just a log. For instance, it would be great to have the ability to see a collection of all files sent by a particular sender, with file versions (with dates and links to according emails) stacked under one file name.
Another suggestion -- view a collection of files in all emails with particular label.
Finally, in some cases it can be helpful to have a local read-only copy of files in a particular collection. It's like one-way read-only Dropbox. This would make automated processing of email attachments so much simpler. (BTW, a product idea, anyone?!).
Interestingly enough, JMAP will make this kind of thing much easier - and we have thought about it in the design! Our Cyrus server already internally keeps a blobId which is a digest of the binary content of each MIME part. Which means you can easily tell that two attachments are identical without reading the contents or guessing based on size and name.
And it's easy to search for messages with file attachments and fetch just the file attachement details via JMAP, making this kind of interface relatively easy to write.
The usual caveats about "we haven't got a particular timeframe in mind" apply - but once JMAP is standard, anyone can write a client to that standard!
That said, I'd still encourage you to think bigger to make a more general system. Of course, the risk of thinking bigger is you may end up with either nothing or a monstrosity no one wants to use. And also you'd likely have to adapt your business model eventually if the bigger picture changes through your actions. So, it's fair to think smaller too.
From a related post I made to the Thunderbird Planning list in April 2017 on what I see as the bigger picture: https://mail.mozilla.org/pipermail/tb-planning/2017-April/00... "My perspective is from wanting to create tools to support a "social semantic desktop" of which email is a subset of the items. Other types of items include news messages, RSS feed messages, chat messages, notification messages, IoT messages, blog posts, pingbacks, saved web pages, commands constructing complex shared documents, VR world update messages, code snippets, notes, and more. Email is a special case of a much larger world of messages. So, for me, a good assumption is that people will eventually have billions of message items comprising terabytes of data in their local (shared-with-email) store. Eventually each user would be part of a global federation of users that together have billions of times that number of messages as a federated semantic web. I understand that such a vision of messaging on that scale is not yet the goal of most people in this discussion right now. But I'd still urge people to think a little beyond redoing Thunderbird as-it-is and try to imagine a personal messaging system that processes, archives, indexes, and recalls vast numbers of items every day in interaction with a user and automated systems as part of a larger federation of similar systems using shared semantic standards."
I've been beating my head against that wall for years with very limited progress (as have others) -- so I can't say it isn't a high risk endeavor with a low probability of success. But the benefit for humanity could potentially be huge if such an effort succeeded. Maybe it might just take a few tweaks to JMAP to generalize it enough to realize some of that big picture?
We also use JMAP-style APIs for Files, for Notes, and for all the extra Preferences and group management stuff in both FastMail and Topicbox.
As related market research, Kolab uses IMAP as a general-purpose data store: https://github.com/pdfernhout/Twirlip2/blob/master/design/ma...
No! It's a trap!
I'm not sure if jmap is all that great (but pretty sure it's in the "all email protocols suck, but jmap socks less..."-bucket ;-)
Someone did think bigger, and the best of them begat lotus notes. And/or the original couchdb. Walk får enough down that path and you come up with distributed executable objects. And while the vision is pretty, it's anything but simple - or something that can survive for 40 odd years. Today, you'd probably get something like urbit. Cool research: yes. Might be useful: yes. Something that'll work well to access email from an iPhone, a mainframe, your openbsd firewall and a windows 10 tablet... Not yet - probably not ever. In fact, even the modest goals of jmap puts it at peril in this case.
Lotus Notes and CouchDB (designed by someone with Notes development experience and related aspirations) are projects that do have quite a bit of alignment with what I am talking about. And they can be useful systems even with their limits. Notes suffers from being proprietary and having warts from its evolution (including how apps are implemented). CouchDB suffers the limits of trying to be both a database and a server at the same time and hitting limitations in both areas. It also unfortunately had a logo that could be misinterpreted as something offensive. That said, I do like CouchDB a lot as an inspiration and seven years ago or so I wrote a small app that runs on top of it.
Thanks for mentioning Urbit. After a brief perusal of their mission and code and a HN article on it ( https://news.ycombinator.com/item?id=8578151 ), it seems to me at first glance (perhaps not doing it justice) like a new flavor of Lisp (but with math limited to increments to produce a functional-ish Flux-like model) crossed with a bunch of techno-utopian aspirations about personalized distributed computing backed with some code and a lot of hand-waving. One can also see the economic model in there to please investors by creating an artificial scarcity of address space that is auctioned off. So it is a set of sense and nonsense it would take a while to sort through -- but so often the challenge in life is doing that for our own beliefs given everyone believes many useful things and a few problematical things. Certainly a lot of things it points to are worthwhile aspirations (like having more control over your own data) whatever one may think of the implementation. Some of its aspirations (the ones I'd tend to agree with more) are reminiscent of the FreedomBox project -- but without FreedomBox's practical grounding in mostly repackaging existing Linux technology.
In practice, these days I'm thinking more about a set of npm packages that make it easy to create your own personal server with a combination of "npm install" commands to add new capabilities (including perhaps a JMAP server for some of your data that could be interpreted as email-like messages). Ideally, that personal server could be federated with servers from other people using some standardized APIs (some of which may be very CouchDB-like). And if people need a unique identity (or several), that could be done by people providing public keys to others as their "identity" and using such keys for signing messages. That's a way to realize some Urbit-like aspirations in a more down-to-Earth way, but leveraging the Node ecosystem instead of how FreedomBox leverages the Linux ecosystem.
Here is a message I sent in 2010 to the Diaspora project along similar lines: "Raising the bar to supporting a Social Semantic Desktop" https://groups.google.com/forum/#!msg/diaspora-dev/TNNpvfFqN... "So, you can take all of my technical comments with a grain of salt in the sense of all that ambition (or whining or paranoia. :-) I acknowledge these suggestions are pretty ambitious, as above, and such ambitions would ask that the Diaspora team learn a lot about emerging ideas and do some really difficult things involving some bleeding-edge semantic stuff beyond the new (and good) stuff it is already doing -- and that it is possible such ambitious and risky efforts might fail for all sorts of reasons. Still, I contacted the Chandler/OSAF project about the Pointrel system semantic approach and they went another more mainstream way and failed anyway (well, it's still ongoing, but it lost its way and most of its funding somewhere along the line). But, I learn more each time, so maybe the above might seem more coherent and approachable than what I sent the Chandler/OSAF people years ago, and that in private, not in public where at least others could learn from it. :-) It's quite reasonable for anyone to say that, for now, a Semantic Web is not what Diaspora is about, as no one can do everything. But somehow, I feel, the really big future for something like Diaspora is going to be integrating semantic web concepts. So, at the very least, if Diaspora was built in such a way as to support future modules in that semantic direction, it might be an even more successful thing that totally and rapidly eclipses Facebook and many other technology platforms as well."
I wish I had more time to pursue these ideas in an open way other than as a hobby taking too much time away from family -- and involving unhealthily spending way too much time on the computer overall on top of regular 9-5 work time. I'm obviously personally invested in these ideas and enjoy programming in this area. But it's also a tough moral choice about spare time given the need for such systems to have a happier singularity factored against the very real costs to any individual/group and a very low probability of success for each specific effort. But collectively, if enough people try (alone or in small groups) in their own ways, likely somebody will succeed -- and then a larger group can build on that success in a stigmergic way. https://en.wikipedia.org/wiki/Stigmergy
Still, while such an effort to create an alternative may indeed be a trap of wasted effort for no results, what we have now with global use of (anti-)social media -- media designed to addict people and provide 24X7 surveillance and control of information flow via a few centralized services -- is a trap we're already collectively in.
Unfortunately, some people who I might otherwise think should know better -- like Paul Jones, UNC professor in the School of Journalism and Mass Communication and the School of Information and Library Science and of ibiblio fame -- seem to promote the centralized social media technology of this trap. They have given up on email due to spam, information overload, and other issues like asynchronicity -- and replaced email by collection of mostly commercial web services rather than try to make something better inspired by email's good aspects including local archiving and local search. See: "UNC professor Paul Jones went from pioneering to protesting email" http://www.dailytarheel.com/article/2014/02/unc-professor-pa...
While centralized services do have some benefits over current email (including editing and moderating), it is not like the centralized services don't have their own new problems -- whether various forms of spam including low-quality duplicate content driven by monetization, having less time for reflection in a torrent messages with immediate response expected in some services, or even new problems where "unfriending" someone is a lot more complicated and stressful than not including someone on an email CC list.
I've posted a few comments over the years on Paul Jones' #noemail blog ( http://www.ibiblio.org/pjones/blog/category/noemail/ ) agreeing with him about email's problems -- while also pointing out obvious new problems with trusting all your communications to a few centralized services and suggesting working towards better alternatives. But those posts seem to have disappeared in what may have been a purge of most comments from that blog. That is an example of one of the perils of entrusting your messages to someone else for publication and archiving -- and a reason to consider a federated architecture as an alternative.
Still hoping he might someday promote a solution that keeps email's many benefits (including as a personal "electronic memory") while addressing the valid concerns he raises about email.
Anyway, stuff I like to think about and discuss and implement. Even though I may be reaching the point where, like many others before me, "The point of being done is not to finish but to get other things done." See: http://www.manifestoproject.it/bre-pettis-and-kio-stark/
And one might then hope for continued co-evolutionary bootstrapping improvement of concepts, artifacts, services, culture, and practices like Doug Engelbart talked about. And we can see that now, like in using the web right here and now on HN to discuss something better.
So, I can totally get where JMAP-as-it-is becoming a big success would move things forward, and we could then use JMAP to discuss the next generation of changes. So, I'm raising the issue of a broader aspiration -- while accepting it might not be best for JMAP right now.
1. https://www-archive.mozilla.org/blue-sky/misc/199805/intertw...
Jamie is obviously a brilliant and humane guy -- and his Lisp roots really show in his design of general purpose systems. Looking at Jamie's website, I was hanging around the CMU Robotics Labs around 1986 when he was too so I wonder if we may have ever talked in passing then?
LISP as a programming mindset encourages such generalized designs about information handling, transformation, and linking -- as does inspirations from early AI research (thinking about thinking). For example there is the book Representation and Meaning by Herbert Simon and Laurent Siklossy from 1972 -- as one of many early AI books. I have that one because Professor Simon was nice enough to give me a copy back then during a few conversations we had in his office. Before that, in high school circa 1980 I did an independent study project using Patrick Winston's book on AI. William Kent's 1978 book "Data and Reality" (the first edition -- the third edition was dumbed down by someone else) is another great inspiration. There is a sort of cross over between AI-inspired design and human-centric "augmenting" design.
An interesting comment on the AI/IA crossover by Brad Neuberg (who worked on Open HyperScope): http://codinginparadise.org/ebooks/html/blog/ia_vs__ai.html "I‘ve traditionally been in the IA camp my whole career; I’ve spent the last year going deep into the AI camp. Traditionally these have been opposite worlds but I’m wondering if there might be some powerful, interesting ways to combine them together. One is humanistic and and design focused, the other tends to be more analytical; the space between them is where I’m interested in exploring in the future."
Back around 1988 I applied to be in the NeXT developer program with a concept for an application (embarrassingly called "Orgi" back then, but I was young and reckless/foolish) where the idea was that any data item could hook up with any other data item (using triples). That application never got acted on until I met Steve Jobs when he gave a talk at Princeton and asked him personally about the developer program. And such ideas came out of my earlier interest in triplestores -- ideas that contributed to the roots of WordNet by my undergraduate advisor at Princeton, George A. Miller started just as I was graduating.
In any case, as I said, "I've been beating my head against that wall for years with very limited progress (as have others)". :-)
Jamie was obviously smart enough to connect with such ideas and move on to a happier life selling beer, pizza, and conversational ambiance. Obviously a much smarter guy than I am. :-) And I did not realize his early interest before, so thanks. I knew of another person who became much happier leaving contentious academia after his PhD and being a professor to run a hardware store. See also Matthew B. Crawford's "The Case for Working with Your Hands" written by someone with a PhD who left the non-profit public policy sector to do motorcycle repair: http://www.nytimes.com/2009/05/24/magazine/24labor-t.html
That said, maybe if Jamie had run Mozilla we would not have seen such a tragic waste of billions of non-profit dollars mostly pissed away -- including by mostly abandoning Thunderbird? Or, maybe his voice would have just been ignored? Hard to day. Innovation is such a complex thing. Some ideas I've collected for supporting innovation at such places: https://github.com/pdfernhout/High-Performance-Organizations...
Doug Engelbart and many people he inspired are others near the root of this "intertwingled" data ideas (including "Open HyperScope" as a successor to "Augment" http://hyperscope.org/ ). Ted Nelson and people he inspired with Xanadu. Alan Kay and Dan Ingalls with Smalltalk and its implications as a thinking tool (Dan ran a family hotel for years afterwards). Ward Cunningham with the Wiki server idea and now the Smallest Federated Wiki idea. Mitch Kapor and the Chandler Project. The recent Dat Project by Max Ogden and others also connects with all this somewhat. There is this great book called "Design and Memory" from 1980 by Peter Huyck and Nellie Kremenak exploring such ideas. And of course "As We may Think" by Vannevar Bush. But the roots go further back though annotated scrolls (and the Talmud) and millennia of libraries -- and even back to the first "Standing Bear" cave paintings (see the book, "The Walking People").
This issue of (re)creating a decentralized alternative based around an email-ish feeling of interlinked locally-searchable information is increasingly important because it potentially is a way to move beyond negative aspects of centralized (anti-)social media.
But, hey, my wife and I have more than 5000 books in our house, so all this might just be my hoarding tendencies trying to justify themselves? :-)
Well, off to an unrelated job. And it is unrelated precisely because most of the big paying players in this space (e.g. Google and Facebook) want to get between people and their data -- and typical startups make deals with investors which limit their ability to do FOSS. Still, I can hope there will eventually be, say, an "Automattic" of this "intertwingling" equivalent of WordPress. I actually had hoped Automattic might itself become such a company, but they sadly seem to now see themselves just as a Wix/Squarespace competitor (e.g. adopting Slack for internal communications -- and not even Mattermost or Matrix.org if they want to let someone else own that space and cooperate with another FOSS system).
At least it is nice to think about these things in my spare time -- although sadly at the cost of other experiences. Maybe it is all just a mistake to use my non-job time like that, like Piotr Solnica suggests: https://twitter.com/_solnic_/status/953578620617920513
As an alternative to mbox/maildir/whatevs I've been toying with a prototype of email storage that works in a content-addressed way with objects very much like how git operates internally.
> Our Cyrus server already internally keeps a blobId which is a digest of the binary content of each MIME part. Which means you can easily tell that two attachments are identical without reading the contents or guessing based on size and name.
This deduplication property then comes naturally from the above design, and not just for attachments but also for email headers and bodies.
Many other things come easily such as rsync "just working" for fetching, or making use of git packs[0]
Sadly deduplication isn't quite that easy - while the binary formats might be the same, MIME is very un-roundtrip-friendly when you start from the MIME structure. There are many ways to encode the same bytes, so if you need to reconstruct the raw bytes (FETCH RFC822) then you need a way to recover those original bytes. I've started work on that a few times, but it's never been worth the effort to make it stable.
When gmail first launched I remember it having such relatively high storage space that my friends and I used it extensively as a music sharing/storage system. An mp3 was just small enough to make it a comfortable email attachment and the storage space was great in an age without infinite cloud hosts.
Huge amounts of gmail's early utility to me was simply the file storage. Lots of people would give you a free email address. Google made a big deal out of the massive space they offered.
I was eventually able to get it by doing "Show Original" in Gmail and copy/pasting the base64-encoded attachment data and decoding it. But it made me nervous about using Gmail as a way to hang on to files, since things I attach now might not be available later.
All these features does gmail already have for years. And yes, people also use gmail for file storage. AMP is a stupid idea, but it doesn't mean they do it instead of other features. The other features are there already.
First, it's a task queue. And you're the worker that processes the tasks. Anything that helps an email service to be that abstraction is very good for users -- prioritizing, postponing, auto-processing, branching, delegating, etc.
Second, it's a log. As the article rightfully says -- an immutable one. Again, anything that helps email be a good log is very good for the user -- search, filtering, archiving, etc.
Third, it's a file storage. This is the most overlooked part, unfortunately. Search, sharing, organizing into collections -- all that would be of great help to users.
Yet, Google pushes that silly AMP :\