BlackForestBoy
We’ve developed Memex to solve for that. It’s an offline first extension for bookmarking and annotating websites, pdfs and youtube videos. Also you can collaboratively curate and discuss them, and it has a mobile app to save and annotate websites. It’s availabe for Chrome/Brave and Firefox (memex.garden)
3min Demo: https://links.memex.garden/3mindemo
Especially we focused on indexing and page load performance.
It may be that you have not touched the indexing preferences (which only index pages that are visited for more than 5 seconds)
Is that the reason, or does it still not work?
I am confused. We never offered lifetime subscriptions. However what we did is give people who supported us between 4 and 5 times the supporter amount in credits they can use to upgrade. We sent an email around to everyone at the end of last year.
(You're the only "Nikolay" in our customer DB, so I gave it a check and you have tons of credits still left)
The reason it was "free" for you at checkout is because the credits were applied.
Hope that clarifies things.
Kind of a bummer that this trended one week too early - next week we'll publish a big release with performance, UX and stability improvements.
(Oli here from WorldBrain.io)
A bit of a timing issue here :) We are about to have a big release with lots of improvements, including an API, performance, UX and bug fixes.
Our API will be served via Storex https://medium.com/@WorldBrain/storexhub-an-offline-first-op...
Wow. Thanks so so so much for taking the time to write this all out. This has been super useful.
I added all your feedbacks to our UX/UI/bug prioritisation board. Some of the things we had already on the radar (like improving the way annotating works) and will be improved very soon.
It already collects some basic interaction data like visit frequency, stay time and scroll %. This data could be used to clean the db a bit.
Other than that there is definitely room to improve to clean out the terms that are captured but not really add value. Its a difficult task though because every page is so differently structured.
What ideas do you have to reduce the number of unhelpful terms? A spontaneous one is to detect the footers of this OutBrain et. all crap and remove them from the HTML before filtering out the words to index.
Right now we are focussing on developing a stable service with better UX and that does the current feature set really well. I'll take up your suggestion so we can think about how we can use them in the upcoming overhaul of the search.
Thanks for your input!
Memex App will be a dedicated app that in its MVP works just like Pocket, except that you can also add notes, lists, favorites and lists when you save content.
In the MVP+ you can also search your knowledge on the go and do annotations on the go.
Thanks for adding so much input to the conversation. :)
Let me address a few points you made:
> A complete P2P system would upend that model, assuming it could be made to work well (the asymmetry of the broadband infrastructure for consumers doesn't help, either).
There have been a few decisions we had to make in the past in order to provide a non-technical user friendly, scalable, affordable and privacy focused product.
One option would be to use newer p2p technologies like IPFS(https://github.com/ipfs/) or Dat(http://datproject.org). Those would provide high decentralisation but have a couple of drawbacks that make it not suitable for our use case. Indeed they are difficult to make work well. 1) The technologies are not ready yet. They all still have significant performance and scalability issues which won't be solved in the next 1-2 years (optimistically) 2) They are unsuitable for private data as they shard your very personal knowledge across nodes you don't know. At least that is how they now work. Private networks are planned but it is still a far way to go. 3) They are not suitable for non-technical users yet
Next option are blockchains. Let's not talk about that :P But seriously who wants to store their personal data like a history on an immutable ledger. Nope not gonna happen.
Next option is using a p2p sync via WebRTC, which we actually do use. Our servers are only there to offer a relay service to pass your message for asynchronous syncing and signalling for synchronous syncing and to punch through your firewalls. The sync messages are end2end encrypted and deleted from our servers when all devices have picked up the message. This approach offers the ability to be much cheaper than whats out there because we don't have to store the data on a cloud constantly.
> Today's system takes control away from the users for the most part, unless they pay extra (ie - to create and maintain a server or whatnot for their own personal content - or not pay, or not pay by giving away other information that the company giving them "free hosting" or whatnot can use).
For both p2p or cloud infrastructure, you won't get around servers. Someone needs to pay for that too. (and we are not even including the development costs for the software). Even if you would use a full p2p system like IPFS and dat, once they become more common there is a need for infrastructure someone maintains. It's probably your ISP that then starts charging more. There is no free lunch. In the end it begs the question if it it so bad that you have to pay a bit for services that really make your life better? I don't think thats a bad thing. We got so used to getting things for free without valuing how they contribute to our lives. Except of course you give away your data, in return for those free services in (implicit) exchange for data, which is not an option for us. We won't rule out that there might be a very consensual relationship between users and us to share data and do some amazing stuff with it, but also let them participate in the fruits. But that is definitely not the default like on most other services. By default your data is always yours.
In our case we will initially offer the syncing service for a premium subscription. The code is all there though, so it can be made self hostable. We either need a committed group of contributors taking this in their hands, or need to make the money first to have the capacity to do that. Either way it's unlikely going to happen immediately, but we definitely want to see it.
> (this seems to be a siloed system, if anything).
However most importantly is to express one of our core values: We think optimising for interoperability is far far more important than decentralisation. We believe if users have the ability to easily move between different providers/silos of a software and take their data and social graph with them a lot of the trust issue we experience today, and hope decentralisation helps, would be solved. If users can migrate to more ethical and privacy focused services easily, it would put an incentive on ethical and trustworthy behaviour and would be able to still use the many advantages of centralised systems (iteration speed, cost efficiency, development convenience, performance).
To protect your privacy, data ownership and freedom to move, we already invested considerable effort.
1) We focus on the software being offline first. This had quite some challenges, among others to get search and storage performant enough in the browser.
2) We build a database and storage layer that will turn into an interoperable datastore for knowledge data that gives you full control over your data (https://github.com/worldbrain/storex). Memex will turn into a light client so you can copy/fork it, adapt it to your needs and use the same database. So you don't even have to migrate anymore.
3) We have a completely different approach to set the economic incentives in our company. We don't raise venture capital so 1) we don't create incentives to lock you into our service and 2) provide you free services that exploit your privacy for the sake of growth: To raise capital we use a model called Steward Ownership, that aligns with the incentives for interoperability. We did that because we believe there needs to be an ecosystem of many "memex"-like tools developed by other people that interoperably work together. More you can read here: https://community.worldbrain.io/t/why-worldbrain-io-does-not... It took us almost 2 years to find money from investors because not taking venture capital was so fundamentally important for our vision. So we had to refuse a lot: https://community.worldbrain.io/t/how-worldbrain-io-tries-to...
I hope that answers many of your questions, and likely spurs many more. Happy to answer them :)
Both projects took the inspiration for the name from the same thing though: https://en.wikipedia.org/wiki/Memex
Ok so the core issue was that the sidebar was popping up in places you didn't want to? Or was it also that the keyboard shortcuts were firing the sidebar to appear in places you didn't want?
> Text search is hard, I know that from mere experiments and demos; adding the "be a browser plugin" to that must make life fun indeed.
Yes, it was/is quite painful, especially in the browser. With its changing APIs, and websites constantly changing and using different technologies, makes browsers a very hostile environment to work with. We can't even do test releases with the chrome/firefox store. So we have to roll out all updates to all users at once. Crazy.
Before we focused on building something to show how our vision could work at the expense of stability. It was in a lot of ways not a good move because we probably could have been financially independent with a little less features that were more stable and better worked out. However our approach also allowed us to show the vision and get bigger amounts of funding. Now is the time to make the current feature set nice to use and people feel like it is worth financially supporting.
It was a trial we added in the last onboarding overhaul 2 weeks ago. Will already be changed in the next update.
> I'd reconsider the landing page hero section design. Don't write with black on darkgray background.
Yeah that is a bug from the last (automatic) wordpress update. Will be fixed soon too. Thanks for making us aware. :)
Yeah indeed. A better implementation for now would be to let people save single posts, or sync with likes, shares, tweets and retweets, so people can search with those facettes.
> Is your code all home-built, or you're using some FT engine compiled to Wasm?
For the search we are using dexie.js, the rest is home-built. Our storage engine is: https://github.com/worldbrain/storex
glad you bring that up :) After we are finished with the current work on making Memex more stable and user friendly, we are working on shareable collections. With that you can share lists of websites, papers, notes and annotations with your peers, or co-curate those collections together.
Please don't feel sorry. :) If something does not work then we deserve those things to be said, also publicly. It's what after all helps us to improve and keeps us accountable.
> "respond when i type" setting several times.
I am confused about that setting. Not aware we have that anywhere? Can you explain a bit more what you mean by that?
> the indexing was off again
That can be, we had some issues with FF permissions for things to be copied to clipboard. We had to manually add that permission to the install process (only for FF). And I missed doing that a few times. (sorry about that). We are about to automate that process in the next releases so that does not happen anymore. I assume what happened is that the extension was turned off because on update it required some of the permissions to be granted again.
Yes! That is indeed a major problem with all knowledge management tools. They tend to not be interoperable enough so you can easily integrate them into your existing workflows. Also they are not built to be adaptive to the individual workflows of people, so you have to wait for the dev's priorities to be high enough to implement your features. You can't do it yourself.
We are also not there yet when it comes to the level of interoperability or flexibility needed. However we started from a fundamentally different angle by changing our economic model and not taking venture capital money: https://community.worldbrain.io/t/why-worldbrain-io-does-not...
In essence what we want to achieve is that you can copy/fork Memex, adapt it to your needs and still use your old data and social connections. Once that transition is complete you'll be able to even use 2 different Memex tools at the same time, both maybe serving different use cases for you.
May I ask what tools you use and how Memex in your ideal world would integrate them? What is the workflow you'd like to implement?
Yes indeed we will be a regular SaaS business, but with a strong focus on privacy, data ownership and interoperability. More can be found here: https://community.worldbrain.io/t/why-worldbrain-io-does-not...
Initially our services like Sync and Backup will only be available for the premium upgrade, but our infrastructure is already built in a way that allows making it self-hostable too. It's just a matter of resources we have available. We very much value your data sovereignty but we first need to make enough money to be able to continue development and have enough time and money to make it self-hostable. It's all a bit more difficult without raising VC capital (see link above).
1. Yes the index is updated every time you visit. It appends the new terms, and keeps all old ones
2. That is a bit more difficult and not as reliable unfortunately. If there is a lazy load on the page it often fails to capture the content, because it starts indexing the page after the initial page load is finished/successful. These are improvements we want to work on a bit later.
3. For about a year's worth of history(~20k pages), without also capturing the screenshots, it needs about 400mb of storage. Indexing performance is still good with 20-25k pages but querying gets slower. So you won't feel the performance on your system with a reasonably fast computer (recommended 8gb of ram, and a dual core with at least 2GHz) We are about to work on performance improvements to make it fast and scalable much beyond that amount and with less resources.
4. Unfortunately we didn't get around to optimise the indexing for CKJ characters, but all latin characters should work fine.
A bummer, sorry you experience troubles.
> Texts highlights are a pain to use (so I don't, even though I love highlighting text while reading)
Why are they a pain to use for you? What would be 1-2 things we could improve that would make it a lot better?
> the search doesn't find anything despite the keyword being in the title of the page
It might be that you have some indexing settings changed. Usually the page is only indexed after visiting it for at least 5 seconds, but you can change that setting
> all the context menu and side bar are doing are unexpectedly hijacking my clicks.
On which pages does this happen to you?
Good news: We finally got a bigger round of funding and are now able to fix those. Until February our main focus is improving release stability through fully integration testing the UI and backend, fixing all major bugs, improvements to UI/UX and the release of the mobile Apps+Sync.
We have been featured a couple of times on HN. This one comes up every time. Gold. <3
Yeah we know it has not been really well functioning in the past few months. Had a lot of bugs.
We finally got a bigger round of funding and are now able to fix those. Until February our main focus is improving release stability through fully integration testing the UI and backend, fixing all major bugs, improvements to UI/UX and the release of the mobile Apps+Sync.
Too bad this post has been upvoted so much now, a few weeks later things would have already been much much better :)
Regarding the bugs you mention I would like to know more if you could help me out here: 1. Which browser and system are you working on? 2. When you say the settings changed back to "intrusive & useless", what do you mean by that? 3. When you say "it never worked well" what were you expecting to be working but it didnt?
Thanks and sorry for the troubles Memex causes.
are you referring to Memex being card based? In which part of Memex would you like to have that drag&drop ability to organise stuff?
Up until now all data is stored on your own device, or in your personal cloud with a file system integration (like a Dropbox folder)
We are about to release our mobile apps for iOS and Android, as well as syncing between devices. Both the extension and mobile app will be offline first and sync happens with an end2end encryption via a relay server that deletes all messages after all devices have been picking them up. That server will in the mid-term be self-hostable too.
Thanks for the feedback. Except it being able to be turned off, how could we make the implementation less intrusive for you?
Memex is not able to do cross-browser/device yet, but we are short before releasing our mobile apps for iOS and Android, as well as syncing between devices. Both the extension and mobile app will be offline first and sync happens with an end2end encryption via a relay server that deletes update message after every device has been picking it up.
At the moment analytics are turnt off, but in general we only do telemetrical analytics. So we only track the way you interact with the software, like the buttons you click, to optimise user flows. We don't ever send anything that is user generated content, like tags, annotations, visited urls or anything that can be considered your personal knowledge.
More you can see here: http://worldbrain.io/privacy