bnewbold

Where are you getting the 200 megawatt number from?

The document you linked says that a large auto assembly plant consumes around 188,000 MWh annually (with regional variation). By my quick math that is less than 22 megawatts baseline load (24/7/365).

There is a mention that natural gas and other fuels being used on-site, are you converting those to MWh equivalent? I'm not as familiar with that conversion, but from a quick online calculator I found it would still be under 75 megawatt for electrical and fuel-equivalent combined.

Thanks for the response Nora.

Because of your blog post I went through the process of setting up a did:web account myself this afternoon, and it was painful. Eg, I found a bug in our Go SDK causing that "deactivated" error (https://github.com/bluesky-social/indigo/pull/1281). I kept notes and will try to get out a blog post and update to 'goat' soon.

We've also been making progress on the architecture and governance of the PLC system. I don't know if those will assuage all concerns with that system immediately, but I do think they are meaningful steps in reducing operational dependency on Bluesky PBC.

fair enough, the did:web flows are not documented even for technical atproto developers, and there needs to be a self-serve way to heal identity/account problems elsewhere in the network (the "burn" problem).

I do think that did:plc provides more pragmatic freedom and control than did:web for most folks, though the calculus might be different for institutions or individuals with a long-term commitment to running their own network services. But did:web should be a functional alternative on principle.

I'm glad that the PDS was easy to get up and running, and that the author was able to find a supportive community on discord.

(I added a reply up-thread here: https://news.ycombinator.com/item?id=45469667)

DNS poisoning is a concern in some situations, but not always.

The common case with at:// strings is to put in the DID in the "authority" slot, not the handle. In that case, whether resolving a did:plc or did:web, there are regular HTTPS hostnames involved, and web PKI. So the DNS poisoning attacks are the same as most web browsing.

If you start from a handle, you do use DNS. If the resolution is HTTPS well-known, then web PKI is there, but TXT records are supported and relatively common.

What we currently recommend is for folks to resolve handles server-side, not on end devices. Or if they do need to resolve locally, use DoH to a trusted provider. This isn't perfect (server hosting environments could still be vulnerable to poisoning), but cuts the attack surface down a lot.

DNSSEC is the current solution to this problem. But we feel like mandating it would be a big friction. We have also had pretty high-profile incidents in our production network caused by DNSSEC problems by third parties. For example, many active US Senators use NAME.senate.gov as a form of identity verification. The senate.gov DNS server uses DNSSEC, and this mostly all worked fine, until their DNSSEC configuration broke, which resulted in dozens of senators showing up in the Bluesky app as "invalid handle". This was a significant enough loss of trust in the DNSSEC ecosystem that we haven't pushed on it since. I think if we saw another application-layer protocol require it, and get successful adoption, we'd definite reconsider.

"A Full-Network Relay for $34/month" https://whtwnd.com/bnewbold.net/3lo7a2a4qxg2l

This one has been popular recently: https://bsky.app/profile/spacecowboy17.bsky.social/feed/for-...

(there are several feeds named "For You"; IIUC this started a couple weeks ago and is based on "likes by people you follow"

Yes, all of those social graph relationships are hinged off a permanent identifier (DID) and everything comes along when accounts migrate PDS instances. Folks can use zeppelin.social from any PDS instance. The DID PLC directory is currently hosted by Bluesky, but the directory can be forked, and did:web identifiers can be used as an alternative (and several independence-minded folks in the network do so).

Migration between servers is so seamless that is causes confusion and doubt that the protocol even supports migration, because there is basically zero in-app visibility of which users are on which server.

Yes, the network continues to work on zeppelin.social if Bluesky servers are down.

If you want a service which indexes every post in the public network, including from folks you don't follow, that is just going to require resources. I think $200/month for a full-network index (as zepplin does) is very reasonable and approachable for organized groups without external funding. Many Mastodon instances cost more than that, and provide a must smaller scope of indexing.

If you want a small scaled down setup for just a small community, which still interoperates with the full network but doesn't have a complete network, there are setups like AppViewLite, which can run on, eg, an old laptop at home: https://github.com/alnkesq/AppViewLite

Personally, I don't think individualist self-hosting is a necessary or helpful goal for indexing the network. Most humans are not interested in spending the time or learning the skills to do this, even if it was as easy as setting up a self-hosted blog with RSS. I think small collectives (orgs, coops, communities, neighborhoods, companies, etc) exist and can fill this role.

Regardless, this is moving the discussion, which was about whether it was possible to decentralize each component the network, not whether it was pragmatic for individuals to self-host the whole thing.

hundreds (thousands?) of users have signed up for Bluesky Social, then moved their accounts to independent hosts. folks can use https://zeppelin.social/ as a totally free-standing bluesky posting experience that interoperates with the full network.

Bluesky Social still clearly dominates the ecosystem, but there is no single component of the system that does not have a open/alternative option for exit.

Do you disagree? Is there a specific centralized component you take issue with?

I agree with the general sentiment here, but don't like the examples. 200 photos per person per year isn't very much! That is all fine.

What really bloats things out is surveillance (video and online behavioral) and logging/tracking/tracing data. Some of this ends up cold, but a lot of it is also warm, for analytics. It bloats CPU/RAM/network, which is pretty resource intensive.

The cost is justified because the margins of big tech companies are so wildly large. I'd argue those profits are mostly because of network effects and rentier behavior, not the actual value in the data being stored. If there was more competition pressure, these systems could be orders of magnitude more efficient without any significant different in value/quality/outcome, or really even productivity.

SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited).

The docs and operational tooling feel a bit janky at first, but they get the job done, and the whole project is surprisingly feature-rich. I've dealt with basic power-outages, hardware-caused data corruption (cheap old SSDs), etc, and it was possible to recover.

In some ways I feel like the surprising thing is that there is such a gap in open source S3 API blob stores. Minio is very simple and great, but is one-file-per-object on disk (great for maybe 90% of use-cases, but not billions of thumbnails). Ceph et al are quite complex. There are a bunch of almost-sort-kinda solutions like base64-encoded bytes in HBase/postgresql/etc, or chunking (like MongoDB), but really you just want to concatenate the bytes like a .tar file, and index in with range requests.

The Wayback Machine's WARC files plus CDX (index files with offset/range) is pretty close.

This is exactly the problem that the Internet Archive created their Scholar project to mitigate (https://scholar.archive.org/about). The https://fatcat.wiki component acts as a dashboard to track preservation of scholarly publications across multiple efforts. There are a bunch of projects in this area, including LOCKSS ("lots of copies keep stuff safe", including some fun/novel uses of cryptography), Scielo and similar regional platforms and archives (primarily outside the US/EU), Pubmed Central, etc. Zenodo (CERN) and figshare end up being an accessible option for some small journals. There are definitely gaps that content falls through and gets lost.

A few folks have mentioned shadow libraries like Sci-Hub. These efforts can play an archival role, but tend to focus on access, which means there is not as much attention on content which is freely available today, but could disappear in the future.

A common dynamic here is that clout and funding flows to globally prestigious publications, and there is a bias against marginal publications. For sure there are many content-farms and scammy publications, but a lot of gems and valuable small publications get bundled in and dismissed.

Some more references and reading for folks interested in this subject: https://guide.fatcat.wiki/bibliography.html

(I helped create Scholar, but don't currently work at Internet Archive)

not as satisfying as a friend's "<name> at at dot dot see see"

This tool is so great for robustly dealing with content in old and poorly formatted HTML. There are a lot of similar tools for extracting "the main text" from free-form HTML, but this was the most reliable in my experience, especially when dealing with web archives containing hand-written HTML back to the 1990s, working with non-English languages, etc.

all public account content is in a "repo", commits to the repo are signed, and the identity resolution mechanism gives anybody the current/active signing key.

the most direct analogy is to signed git commits. this is an intentional design decision compared to signing individual messages/posts/etc. A "proof" for a single record in the repo is the commit, the record, and the chain of merkle tree nodes connecting the two.

some quick thoughts/notes (I am on the bluesky team, but this isn't an official policy statement):

- content on bluesky is public, but we have not set expectations/comms around that well yet, and this dump may be a surprise to some existing accounts. where exactly bluesky falls on the spectrum from "congressional register (immutable)" to "public web" to "public IRC or discord room" to "private signal group" is still being worked out, but probably closest to "public web"

- the protocol supports both "deletions" (retaining history), and "purge" (aka "rebase") to remove all not-current content. this isn't exposed via UI yet and accounts have not had the chance to purge old deletions

- the federation protocol and unified firehose should make it possible for third parties to maintain a live mirror of the entire corpus. importantly, it will be easy (or at least "easier") to respect intents w/r/t deletions when done this way, compared to dumps

- obviously neither "deletion" nor "purge" can perfectly remove content from 3rd party dumps and infra, or from hostile parties. but it does signal user intent clearly, and we expect as a norm that third parties will respect that intent. ADS-B, robots.txt, CC licensing are related to these norms, though all unique. right-to-be-forgotten, archiving, re-use licensing, use in ML training, commercial/non-profit reuse, search indexing, etc, are all on our radar

- blobs/images are not included in this corpus

- this specific corpus does not (I assume) include our important "label" moderation metadata. at least for our (Bluesky) core moderation decisions, that information will be public

- private/group content is not yet part of protocol. eg, no built-in mechanism for DMs or follower-only posts. we will probably do those eventually, but it will be basically a whole separate protocol, not a bolt-on to existing stuff. wildly different privacy/security concerns with non-public content

- there are some other cool projects, like https://bsky.jazco.dev/, working with the full social graph, pulled via public API

dropped domains will be a bit of a UX problem.

the human root of identity is the handle (memorable/recognizable), but the real identity root for the account is a DID. which is a pretty open/wild spec, so we mostly use did:plc ("placeholder"), which is a self-authenticating tricky tricky.

withing the protocol and applications, everything under the hood works via DID references, not handle references. a domain handle works by pointing at a DID, but does not control the actual DID ("DID document"). so any old users/followers/href will still be attached to the "old account". and it would be possible for the old account to recovery and set up a new handle.

but the superficial bits (anchor text), and human identity for new lookups, are attached to the handle, and could get pointed to a new DID, or just not resolve. that would be messy.

You can point your web browser to https://staging.bsky.app

The team is too small to develop and maintain even an electron desktop app or something like that wrapping the existing react-native-web SPA. But this leaves space for fancy third-party apps, or integration into multi-protocol desktop apps, as needed.

I don't really understand your question.

Maybe it helps to clarify that the concept of a handle is totally distinct from the "Personal Data Server" or PDS. You look up the PDS for a handle by resolving the DID first (via DNS TXT or HTTP fallback), then get a DID document, which has a URL to the PDS. This is a bit confusing and multiple steps, but is what makes the handles so easy to use. It is totally fine to have a cool domain to use as a handle and just not have a webserver associated with it.

If you pointed the domain with a CNAME to a particular service, you would need to update that every time you change hosting providers. Which is maybe not often, but it would be disruptive and potentially poor experience because of DNS propagation issues.

For a hypothetical brand like google.com, look up the TXT records. There are a bunch of verifications for various services. This is a relatively standard mechanism. TTL and caching are built-in to DNS, or you can punch through with a recursive query when needed.

atproto handle resolution tries DNS TXT first, then will try an HTTP endpoint. the endpoint is not a ".well-known", it is an "/xrpc/"-prefixed endpoint with a query parameter, which is workable on some static hosts (like github pages), but not others.

not well documented yet. may try to do a ".well-known", but don't want a sprawling set of options with complex priority levels and caching behaviors.

As a small nit, André Staltz is not directly affiliated with Bluesky. Paul and a couple other folks on the team have a track record with similar open projects.

No, the Bluesky app does not ask for your contacts.

Somebody else could build an app for Bluesky that did ask, the same as somebody could in theory build an RSS feed reader that asks for contacts.

The content in scholar.archive.org has been indexed in to Google Scholar (and other indices are likewise welcome to crawl the sitemap). There was some content "only" in scholar.archive.org, but now it should basically all be in Google Scholar. We haven't gotten around to describing this publicly, but it was an explicit decision and partnership between the organizations.

Indeed scholar.archive.org does not currently use citation count in search rankings. We have a decent citation graph, which we are working to expose in scholar (it is visible in fatcat.wiki today). Would probably only ever use citation count as a weak boost in search rankings (eg, "any citations at all", "more than 25 citations" as boosts, nothing beyond that), don't want to create too strong a feedback loop influencing future citations.

scholar.archive.org specifically was partially funded by the Mellon Foundation (and partially through donations and other service revenue). IA overall has diverse funding, including grants and service revenue from the USA (Library of Congress, IMLS, etc); other national governments (paid crawl services); foundation grants; universities and libraries (crawl, preservation, and digitization services); and of course general donations. The last category of course has the fewest strings and lets us pursue new projects which might be hard to get traditional funding for. Remember that the whole premise of web archiving was considered radical and quixotic at the beginning!

(source: I work at IA on scholar)

This seems like a good contribution. It would great if one of these "lists of blessed crates" itself becomes blessed and sticks around! This list seems like it was created by somebody with "good taste", and I enjoy the amount of description.

https://lib.rs has some basic features to make it faster to sniff test various crates. Eg, I like being able to see a summary of the authors, and things like "maintained by RustCrypto team" just looking at the summary. My suspicion is that some concept of graph centrality (perhaps via authors?), and perhaps counting applications which aren't usually distributed via cargo (this contributing to crates.io downloads), could both yield a better popularity metric. But the trend stuff is pretty reasonable already.

Some related efforts at documenting pseudo-official crates (in contrast to gigantic "awesome" lists) are the "areweX yet" pages (https://www.arewewebyet.org/), the Rust Cookbook (https://rust-lang-nursery.github.io/rust-cookbook/), and the Rust Lang Nursery (https://github.com/rust-lang-nursery). Here is an ancient github issue from way back in 2014 discussing the idea of "official" crates: https://github.com/rust-lang/cargo/issues/934

I'm really curious to see if efforts like this, and from computational augmentation and automation more generally, will yield progress on symbolic modeling of large natural systems.

There is a segment in Adam Curtis's "All Watched Over By Machines of Loving Grace" which describes scientists trying to model the ecology of a prairie. As described by Curtis, the more data they collected and more complex their model, the worse the predictions of the model became. It feels like the lessons of the past couple generations (many decades) have been that symbolic models don't compose together easily; symbolic analysis only works in simple systems and has hit diminishing returns; numeric and ML methods work well enough; etc. I'm curious if better tooling (augmentation and/or automation) can push through some of these challenges and yield real understanding. Or if there are some fundamental scaling problems that can not be overcome.

It is true that some might need to be done manually, but Google Scholar shows that it can be done, with some level of accuracy, via HTML and PDF scraping. PIDs and more formalized metadata make things much easier. But Google Scholar did result in pressure on platforms/publishers/repositories to put at least minimal metadata in HTML meta tags, and this can be machine-extracted. And there is a ton of content and metadata available via OAI-PMH. Neither of these technologies cost anything to publishers on the margin, once they get them implemented, and many have to reap the discovery benefits of large search indices.

Unpaywall is based on Crossref DOIs (one-to-one records), and adds information about publicly accessible versions of each work. In theory publishers can register metadata about whether articles are OA or not with Crossref, but the quality and coverage of this metadata is poor in general.

Unpaywall will both check if articles are actually available from the publisher (by following the DOI and parsing the landing page), and by looking for other versions elsewhere on the web (eg, a pre-print). It is simple in theory, but doing this reliably for millions of DOIs from thousands of publishers is a lot of work!

Hi, i'm the maintainer of scholar.archive.org.

It looks like we are missing a bunch of your public papers, such as those published here: http://park.itc.u-tokyo.ac.jp/eigo/publication_en.html

Both unpaywall and scholar.archive.org work best with papers that have persistent identifiers like DOIs, PMIDs, DOAJ article ids, or dblb records. Unpaywall currently works with Crossref DOIs exclusively.

With scholar.archive.org (and fatcat.wiki, the backing catalog), it is possible to submit metadata records directly, but it can be laborious and would be better for everybody if this process was automated.

Processing OAI-PMH feeds or extracting bibliographic metadata from HTML metadata would probably improve our coverage, and we are hoping to roll out that kind of scraping eventually. But it has been a challenge to clean and de-duplicate metadata at that scale.