Integrating Standard Site Into Bluesky

[estrattonbailey]

The Bluesky team is working on integrating site.standard.* lexicons. For people using the Bluesky app, this means that when they come across a link published with a Standard.site record, they’ll get an enhanced preview. For developers and publishers building on atproto, we hope this provides a bit more encouragement to use this exciting community-built feature in your own sites.

The work will be broken up into phases. The first, to be released in the next few days, will detect Standard Site records present on shared URLs, and use the data contained within those records to display enhanced link cards within the Bluesky app.

Future phases will integrate Standard Site more deeply, and further showcase the incredible ecosystem growing from this community-led effort.

Enhanced post embeds

Bluesky has begun ingesting all site.standard.* records into our database, and initial appview support landed here in #4915. In that PR, you’ll see that we’ve added a new field to the app.bsky.embed.external post embed record: associatedRefs. These refs are com.atproto.repo.strongRefs that point to the site.standard.document and the site.standard.publication records, if available.

When hydrating feeds, we’ll use this associatedRefs array to fetch the raw Standard Site records from our database. The data contained in these records will be used to construct the new fields on app.bsky.embed.external#viewExternal, which also landed in #4915.

Publishers and other Atmosphere clients that wish to create Bluesky posts for their users with support for the enhanced Standard Site embed should write external embeds that look like this example from the atproto.com blog:

{
  $type: 'app.bsky.embed.external',
  external: {
    uri: 'https://atproto.com/blog/indexing-standard-site',
    title: 'Indexing Standard Site',
    description: 'This guest post from Steve Simkins, creator of Sequoia and docs.surf, outlines the strategy he used to index standard.site records.',
    thumb: Blob,
    associatedRefs: [
      {
        uri: 'at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/site.standard.document/3mji7laev2i2v',
        cid: 'bafyreiav7tayirff3dyoducbjo2xrakjwuwux7keypjaikzr5oyn3euzc4',
      },
      {
        uri: 'at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/site.standard.publication/3memeaezix223',
        cid: 'bafyreigpgh2c7uvtnjdwnjua4f6gu4dsr4g4pjh2qhmmactp72yunpdz7e',
      },
    ]
  },
}

Referencing Standard Site records on webpages

Following the pattern established by the Standard Site Verification docs, articles should include a <link /> meta tag that references the site.standard.document URI of that article. Here’s an example from the same atproto.com blog post:

<link rel='site.standard.document' href='at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/site.standard.document/3mji7laev2i2v' />

Additionally, the Bluesky team has met with the Standard Site core community team to discuss the addition of a site.standard.publication URI reference on article pages as well, and they were amenable. This helps us resolve the data more performantly, but we will still support auto-resolution of publications for backwards compatibility.

However, in order to support linking to a publication directly (not a single article), we require platforms to add a link site.standard.publication link tag on these pages as well. This is not currently covered by the Standard Site docs.

In summary, publishers and platforms that wish for their links to appear using the enhanced link card in the Bluesky app should embed their article pages with link tags like:

<link rel='site.standard.document' href='at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/site.standard.document/3mji7laev2i2v' />
<link rel='site.standard.publication' href='at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/site.standard.publication/3memeaezix223' />

And embed their publication home pages with a link tag like:

<link rel='site.standard.publication' href='at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/site.standard.publication/3memeaezix223' />

Without the addition of this <link /> tag on publication home pages e.g. toni.org/blog, Bluesky will be unable to render the enhanced link card.

Verification

Records are validated according to guidelines found in the Standard Site Verification docs. Records that fail to validate will not be embedded in Bluesky posts, and will therefore not appear as enhanced link cards.

Additional integration notes

• If textContent is populated, we'll use that to calculate and display an estimated reading time.
• If publication.icon is not populated, we will be unable to display an avatar next to the publication information in link cards. We may fall back to the publication.basicTheme colors and the first letter of the publication name.

Testing

To test, check out https://main.bsky.dev and paste a link into the post composer. No need to post at all, the preview in the composer should show the enhanced embed and give you a sense of if everything is looking good.

Here's where our UI is atm for each currently supported case: article without corresponding publication, article with corresponding publication, and publication only.

Not included in the initial release

Although we ingest all versions of Standard Site records from the firehose, updates to these records such as headline edits will not be reflected in posts that already embedded the associatedRefs for these records. This is to avoid what we call the “puppy problem”: someone reposts an article titled “I love puppies” and later finds the headline has been updated to “I hate puppies”.

More moderation work is necessary before we’re ready to support live-updating of data. We plan to work towards that in the near future.

[heaths]

Sequoia - and perhaps others - currently creates a ref from a Bsky post to the site.standard.document. Should it stop and only do associatedRefs? Or do both for older implementations of the Bsky AppView?

[plttn]

I'm just reading through this for the first time, but I think the intent of the new associatedRefs is for the embed on the post itself, and doesn't change the reference from site.standard.document to the Bluesky post, so no changes needed to Sequoia?

[estrattonbailey]

Correct, bskyPostRef on the SS lexicons is a separate thing. Here's a post with correctly applied associatedRefs https://bsky.app/profile/test.esb.lol/post/3mmexpn42dk27

[heaths]

Ah, I see: https://pds.ls/at://did:plc:dpajgwmnecpdyjyqzjzm6bnb/app.bsky.feed.post/3mmexpn42dk27

My mistake. I misunderstood that this was a separate record type. Currently we're just embedding e.g., https://pds.ls/at://did:plc:tg3tb5wukiml4xmxml6qm637/app.bsky.feed.post/3mlx6vj5v2b2q which merely has a backlink of the site.standard.document: https://pds.ls/at://did:plc:tg3tb5wukiml4xmxml6qm637/site.standard.document/3mlx6vipoaw2x

Basically, this is to go the other direction.

[estrattonbailey]

Yep the uri is correctly pointing to the web URL, just need the associatedRefs on the embed. Looks good on my end when shared from the Bluesky app. https://bsky.app/profile/test.esb.lol/post/3mmfhnkl37c2t

Also, if you populate textContent we can infer an estimated reading time, if you like. That'll show next to the date.

[SARAsBooks]

This is awesome. I'm just wrapping up some changes to my blog; so I'll make sure I'm set for this. I'm wondering if we'll one day be able to have a feed in the bsky app that renders these embeds for documents from publications in the bsky pds site.standard.graph.subscription for the user, potentially even bringing that into the Following feed.

In the meantime, what if in this enhanced post embed we had a Subscribe button to add the publication to site.standard.graph.subscription? or even just a visual indication if you are subscribed.

[estrattonbailey]

Great ideas! These are on our short list. We'll follow up soon with some more discussion with the community around this. Look out for a blog post and post thread later this week ✌️

[knotbin]

i think it would be great if the subscribe button wasn't hardcoded to leaflet, pckt, offprint, etc. and instead just made the site.standard.graph.subscription record without leaving the app.

This would make it so subscribe button works universally and independent blogs don't have to implement oauth and a subscribe button themselves, while readers can still subscribe to them so they show up in subscription feeds like leaflet and pckt.

[estrattonbailey]

Alex shared our current thoughts on this https://bsky.app/profile/alexbenzer.com/post/3mnd5p54gbs2i

[blowdart]

In the lexicon we have associatedRefs and associatedProfiles

In my experimenting I was a little surprised to see, with your test records, that associatedProfiles had duplicated information. Is the intent here that there must be an associated profile for each associated refs, and that the order matters? e.g. associatedRefs[0] has the associated profile of associatedProfiles[0]?

[blackmichael]

@blowdart this is actually a bug, we'll get this fixed soon! In the near future you can expect a response to look like

      "associatedProfiles": [
        {
          "did": "did:plc:og5t4xeyapx5fwjx5hnr2c56",
          "handle": "michaelblack.xyz",
          "displayName": "Michael Black",
          "avatar": "https://cdn.bsky.app/img/avatar/plain/did:plc:og5t4xeyapx5fwjx5hnr2c56/bafkreifqgfqhasx4px2fwvz7yp5idd474drwg2yvyfjtk5vz56ta3rtthm",
          "associated": {
            "chat": {
              "allowIncoming": "none"
            },
            "activitySubscription": {
              "allowSubscriptions": "followers"
            }
          },
          "labels": [],
          "createdAt": "2024-03-20T17:22:47.257Z"
        }
      ],
      "associatedRefs": [
        {
          "uri": "at://did:plc:og5t4xeyapx5fwjx5hnr2c56/site.standard.document/3mkczur2yedsr",
          "cid": "bafyreihpxwixpbqtpweghndpapfyiozmpfalfhguesozg4tt4hrblw2sva",
          "$type": "com.atproto.repo.strongRef"
        },
        {
          "uri": "at://did:plc:og5t4xeyapx5fwjx5hnr2c56/site.standard.publication/3mjimbfegzz73",
          "cid": "bafyreifra4vf6lzgj3wi6iyliwvlobpui3q2ie6glsdlyq77v3obmi236a",
          "$type": "com.atproto.repo.strongRef"
        }
      ]

[smart-knowledge-systems]

This is working for at://did:plc:i2fgba5nignuw4nccml33wjp/site.standard.document/3mm6alawett2r / https://smart-knowledge-systems.com/blog/the-case-for-at-you, but the estimated reading time is 1m. This is because I added my full post as markdown in .content (with at.markpub.text) and truncated .textContent, I suppose. I guess I should include the full plain text in textContent?

[smart-knowledge-systems]

nvm, I reread

If textContent is populated, we'll use that to calculate and display an estimated reading time.

[heaths]

If we update the embed and the CID changes, will Bluesky re-render? Wondering if https://tangled.org/stevedylan.dev/sequoia/issues/38 is worth doing.

[estrattonbailey]

Great question. If I understand correctly, you're considering editing the post record itself to update the CID. I would recommend you not do this.

One reason is as you noted, the post aggregations break, as does some of our reply threading logic. If/when we get around to properly supported editing posts, we'll need to fix that, but I don't have a timeline for that.

But the better reason not to edit the post is: we eventually plan to pull the latest data by the URI and ignore the CID. So any posts with properly populated associatedRefs will automatically get the latest record data at read-time. The CID is there as a reference to the data that was embedded at write-time. Moderation tooling will need to take this into account, to diff between what the user shared and what the records were updated to. As mentioned in the original post, this is to avoid the "puppy problem".

We need to sort out the moderation story there before we move to pulling fresh data at read-time. We'll keep the community updated as we progress.

[lemma-pub]

I'm assuming the Bluesky crawler doesn't execute javascript so the link tags must be injected server side, correct?

[estrattonbailey]

Yep that's correct

[lemma-pub]

I'm thinking for client-side apps/SPAs one could set up an API endpoint specifically for returning an html shell with the link tags that the Bluesky crawler expects plus a meta http-equiv="refresh" tag with the URL of the actual article that the browser will render.

Does the crawler put any restrictions like the origin of the link being shared must match the origin of the actual article URL that the SPA will render?