News for hackers. Stuff that matters.

stdcli

You can use your own server to host a gaia hub with docker compuse up: https://docs.blockstack.org/storage/overview.html , or use a one of the cloud AMIs we are rolling out here: https://docs.blockstack.org/storage/overview.html

Making user owned storage user friendly for people with non sys admin skills is a challenge, but something we are working towards.

because our gaia hubs are associated with user's ids, we are also working on automating SSL as much as possible for individual users as well. This is another technical challenge that makes it difficult for the average person to set up their own trusted environment where they control their own data.

https://docs.blockstack.org/storage/overview.html. While we are rolling out AMIs that auto launch gaia hubs, and hosting some to make it user firendly for people with non sys admin skills, you can docker-compose up here: https://github.com/blockstack/gaia/blob/master/hub/README.md on a machine you own or trust.

We are working to make doing even this as user friendly as possible.

You can use gaia hubs, which are user owned stores, to host your data wherever you like. Gaia focuses on user owned data, and leaves the work of network consensus and replication to the identity associated with your gaia hub: https://docs.blockstack.org/storage/overview.html where the identity is defined here: https://docs.blockstack.org/core/naming/introduction.html

We have an Amazon EC2 AMI, and are working on others, but the idea is you could bootstrap the docker-compose on any VM you like, or a rasberry pi if you want even: https://github.com/blockstack/gaia/blob/master/hub/README.md

There are IPFS driver requests and now requests for drivers to support privatestorage by Least Authority as well, if you also want to replicate your data temporarily across some nodal network.

While gaia fundamentally does not require using the comprehensive Blockstack API, we are working on tutorials to abstract the use of only gaia without Blockstack. They are designed to be functional independent of each other, in the same way people can use Blockstack authentication without gaia, the reverse can be true: https://docs.blockstack.org/storage/overview.html

Currently, I want it to be even easier than just bootstrapping a docker-compose in gaia for users to host on their own machine, or rasberry pi or what have you. We are working on that as well as cloud hosted solutions.

I would like for people to be able to launch a vm with a preinstalled image locally on their own machine, not just google cloud, amazon, Digital Ocean etc. The groundwork for a secure and minimal VM is mostly in place. We need to set up more instructions for this but feel free to launch the docker-compose and give it a whirl in your environment of choice if you don't want any of the cloud AMI's we currently offer.

Absolutely. We will be in touch shortly. In the mean time, you can check out our forum: https://forum.blockstack.org/ or hit is up on our slack.
Re:What causes Ruby memory bloat? on Thursday March 14, 2019 @05:09PM (#19393737)
When you GC.start() you are probably forcing a round of garbage collection. I assume it is slow to run GC.start(), but after gives you some runtime advantages, but only for a bit (until you call it again), but then the heap fills up again. You can tune the RUBY_GC_HEAP_GROWTH_FACTOR in GC.stat after initializing and load testing a few times to avoid having to call GC.start() and manually force garbage collections by allocating enough space for the system to run its own processes, while minimizing initializing unutilized memory or having too small of a heap size, which will trigger too many garbage collection runs, both requiring expensively slow kernel system calls.
Re:What causes Ruby memory bloat? on Thursday March 14, 2019 @04:54PM (#19393607)
I wrote a bit about this here, with some small GC.stat hacks to manage it a tad, when I talk about how ruby has it's own heap and manages garbage collection lower down in this post: https://ginxh.io/2018-04-18/high-performance-priority-queues...

Even fluentd had this issue with a default heap size lower than what was needed to initialize itself, causing hiccups, until the GC.stat RUBY_GC_HEAP_GROWTH_FACTOR was tuned in the source code.

Ruby memory bloat is everywhere. Being familiar with gc.stat and being able to tune ruby applications as you test them is a good habit to have if you develop or work with ruby based tools.

Julia Evans, an SRE at Stripe took a sabbatical to work on a ruby memory profiler. Her updates are here: https://jvns.ca/categories/ruby-profiler/

This is a great blog post, it's what I would have wanted to know if I went a bit deeper into this. Good stuff.

Blockstack Engineer here. You do not need to download a browser to use Blockstack. We also have a working mobile version, and multiple apps built on the Blockstack API that have their own ios and android apps. You can check out Blockstack without downloading the browser here: https://browser.blockstack.org

If it is your first time, you will be asked to create an identity on Blockstack: https://docs.blockstack.org/core/naming/introduction.html

Each instance of the "browser" whether on desktop or phone, requires a unique password, associated with the id. The id is stored on the Blockchain, the data is stored in your own gaia hub which you can host wherever you like: https://docs.blockstack.org/storage/overview.html

In regards to your comment about "Kademlia or other DHT": Blockstack previously used Kademlia, but iterated on an improvement to Kademlia by creating Atlas which is what we use at Blockstack. More here: https://github.com/blockstack/atlas

and the specific reasons about why Atlas was built in lieu of Kademlia here: https://blog.blockstack.org/blockstack-core-v0-14-release/

Together, Atlas, BNS, and gaia create the foundation upon which the Blockstack API is built on.

You can build applications using the Blockstack API here: https://docs.blockstack.org/develop/zero_to_dapp_1.html

If you want to explore the ecosystem of existing decentralised applications on Blockstack and other decentralised solutions, you can see that here: https://app.co/

You can read more about how identity, Atlas, and user owned storage works here: https://blockstack.org/whitepaper.pdf and maybe poke around here a bit: https://core.blockstack.org/

Oh yes, also you can download the browser on your desktop if you like.

Re:The art of writing eBPF programs on Wednesday February 27, 2019 @08:12PM (#19268271)
I logged in just to say this: I love the sysdig engineering blog. Please don't stop writing blogposts. sysdig and clouflare are becoming my favorite engineering blogs.
The idea is that gaia hubs are owned by the users.

There are two aspects of this implementation:

1. The Blockstack naming system is used to associate an immutable user identity with the routing path to their storage, for reads and writes.

For an application to store data in the gaia hub, they must sign a challenge text proving they have been whitelisted to write that data, but the data is associated with the users gaia hub.

If the user leaves the platform, all of their data is with them in their gaia hub, which can be used by another app (maybe a competing blogging app for example?).

2. Users host their own gaia hubs.

Right now, the barrier is high for people hosting their own gaia hubs, so Blockstack provides a default one that is associated with their identity, so the average user can focus on using the applications on Blockstack and less on the system administration like overhead in setting up a gaia hub.

However, this user onboarding default does not override the ownership the user has in that their id is the one whitelisting applications and allowing applications to store data, which they can then own and use in other places, or disconnect their gaia hub and simply keep the data as their own personal HD with user addressable content even if they did not want to use it anywhere else, with other apps, etc.

Users can host their own gaia hubs. We are working on migrations, images available on platforms and publishing instructions for how to utilise DO spaces and S3 in addition to the driver functionality that exists now for things like Azure blob storage and S3 buckets.

The newer development is focused around automating the driver with local disk so the same set up can be ported to images on the cloud as well as users hosting their own gaia hubs on a server somewhere. For example, many of our enthusiasts are asking how to host them on a rasberry pi.

Let me know if this answers your question. I would also love feedback or ideas in general.

By the end of this month we will have instructions on how to host your own gaia hub in DO spaces (on Digital Ocean) using the develop branch of gaia in our Blockstack docs located here:

https://docs.blockstack.org/

So stay tuned.

I personally find Digital Ocean to be the most user friendly, so we are trying to lower barriers to make it as easy as possible to roll out gaia hubs by providing instructions for these kind of platforms.

We are also rolling out instructions and images for the major cloud hosting platforms. If you have any preferences for a particular platform you would like supported or any other feedback, let me know.

Currently on master branch, gaia supports the following drivers:

Amazon S3, Azure Blob Storage, Local disk (you must set up static web-hosting to point at this driver), and Google Cloud Storage

You can learn more about how to set up your own gaia hub here: https://github.com/blockstack/gaia/blob/master/hub/README.md

Thank you for that feedback. You can find out more here: https://github.com/blockstack/gaia

We are looking to host a gaia website in the future to make it easier to find and easier for users to host gaia hubs in general.

Let me know if you have any feedback about gaia in particular.

The nice thing about Blockstack is it is not dependent on any one Blockchain. It can be ported to other blockchains and has been previously. Right now it sits on top of bitcoin as it is very hard to perform a 51% attack.

if Bitcoin was under 51% attack, then we would simply accept the name state on the best fork. To learn more about how we would handle this case, you can look here: https://forum.blockstack.org/t/bitcoin-network-disruption/13...

Blockstack Core implements BNS and Atlas, the storage routing system for Gaia. Blockstack Core nodes form the backbone of the Blockstack network. Each node indexes the Bitcoin blockchain and maintains a full replica of all names, public keys, and storage routing information. This makes the Blockstack network particularly resilient to node failure---applications only need to talk to a single Blockstack Core node to work, and a new or recovering node can quickly reconstruct all of its missing state from its peers.

All Atlas nodes lie on top maintain a 100% state replica, and they organize into an unstructured overlay network. The unstructured approach is easier to implement, has no overhead for maintaining routing structure and is resilient against targeted node attacks. When a new Atlas node boots up, it first gets the index of all data keys and hashes of values stored in the blockchain. After getting the index, Atlas nodes talk to their peers to fetch key/value pairs they dont have. The Atlas network implements a K-regular random graph.

Let us know if you have any more questions.

Blockstack is an architecture which supports gaia storage, wherein the blockstack api supports routing the Blockstack identity via the BNS naming system to the users gaia hub, where they own and store their own data.

Gaia is not necessarily dependent on Blockstack, but it was designed to be modularized within this design, so users can control where their data is stored, which is not necessarily the case with IPFS.

It is pretty darn cool :)

Let us know if you have any more questions.

IPFS is a bit different than gaia.

The following are some functionalities unique to gaia, without the unpredictable lookup performances associated with ipfs.

With gaia, you can control where the data is stored, on your own server, a cloud hosting provider of your choice, etc.

Data is read write.

Writes permissions can be delegated.

Gaia can be used with an API where developers, like the developers of Graphite, can build applications on top of it without explicitly dealing with a blockchain using traditional knowledge of javascript or other front end languages and frameworks.

Listing permissions can be delegated.

Data is globally addressable.

It supports multiple backends natively, which supports users choosing where their data is hosted.

In IPFS, data may be initially hosted on just your server, but duplicated elsewhere when a call is made to it.

Like IPFS, it does not need a cryptocurrency to work, and space is reclaimed when data is deleted.

That being said, IPFS is very popular. We are working on ipfs drivers for gaia along with others. For now, we have prioritized drivers for cloud hosting providers to help users gain more control of where their data is stored and hosted.

You own your own data in a "gaia hub" where apps have to have permission to write this data to your hub, permitted you give it to them. This way, if you ever leave graphite, or any other app in the eco system, you still have your data in the gaia hub.

You can use the gaia cli to list all the files you have.

You can host your own gaia hub, or have a default hosted configuration, but still it is associated with your username/identity.

We are working on gaia hub migrations so you can easily migrate data from one place to the other. Soon we will have images to auto launch on most cloud hosting platforms. The docker-compose files should allow for setup on a local server as well. I am prioritizing users hosting gaia hubs on their own servers over any cloud host provider.

Let me know if you have any more questions.

But yes, the entire idea is centered around #2.

Hi, Blockstack engineer here. Thank you for the feedback. Can you tell me what browser you are using?