I have deeply mixed feelings about #ActivityPub's adoption of JSON-LD, as someone who's spent way too long dealing with it while building #Fedify.
-
@silverpill personally I feel like the various activity/object signing methods that get used in recent FEPs are more egregious from a size point of view, when the in spec behaviour for obtaining canonical versions of a resource is to fetch them from their server, instead of relying on random object signing that introduces so much more friction.
@mariusor@metalhead.club I thought the whole point of signing objects or attaching proofs (none of which I do, mind you) are precisely to save the need to make a new request, which comes with its own overhead.
The good thing is fetching from canonical source will never go out of style.
Aside, it seems like I'm only getting Marius's posts, not silverpills. Makes for an interesting one-sided exchange
-
@hongminhee can you point me to the parser you use for fedify?
One of my long term plans for GoActivityPub is to built a code generation tool based on contexts and I would need some prior art to see what's important in parsing JSON-LD and RDF.
@mariusor@metalhead.club It's barely documented, but has worked well so far!
fedify/packages/vocab-tools at main · fedify-dev/fedify
ActivityPub server framework in TypeScript. Contribute to fedify-dev/fedify development by creating an account on GitHub.
GitHub (github.com)
-
@mariusor@metalhead.club I thought the whole point of signing objects or attaching proofs (none of which I do, mind you) are precisely to save the need to make a new request, which comes with its own overhead.
The good thing is fetching from canonical source will never go out of style.
Aside, it seems like I'm only getting Marius's posts, not silverpills. Makes for an interesting one-sided exchange
> to save the need to make a new request
@julian probably. But then there's Mastodon that treats so many activities as transient, therefore unfetcheable, which I think is what made the object signing an actual necessity. And outside of the happy path where the actor that generated the object is already known to the server that receives it, there's still the need to fetch their key, so there's no savings for 10-20% (number out of my butt) of activities... As you can probably tell, to me the frictions introduced by signatures are not a good enough tradeoff to effecting one more request.
-
@silverpill lol, that's simply madness to me. See the sibling reply to Julian why I think signatures, which is what I imagine you mean by "authenticated" are an unnecessary contrievance.
I meant "data object" in this context as the end-result binary data type that your application deals with, which for my preference, needs to match the structure of the incoming payload as closely as possible.
-
@silverpill personally I feel like the various activity/object signing methods that get used in recent FEPs are more egregious from a size point of view, when the in spec behaviour for obtaining canonical versions of a resource is to fetch them from their server, instead of relying on random object signing that introduces so much more friction.
-
I have deeply mixed feelings about #ActivityPub's adoption of JSON-LD, as someone who's spent way too long dealing with it while building #Fedify.
Part of me wishes it had never happened. A lot of developers jump into ActivityPub development without really understanding JSON-LD, and honestly, can you blame them? The result is a growing number of implementations producing technically invalid JSON-LD. It works, sort of, because everyone's just pattern-matching against what Mastodon does, but it's not correct. And even developers who do take the time to understand JSON-LD often end up hardcoding their documents anyway, because proper JSON-LD processor libraries simply don't exist for many languages. No safety net, no validation, just vibes and hoping you got the
@contextright. Naturally, mistakes creep in.But then the other part of me thinks: well, we're stuck with JSON-LD now. There's no going back. So wouldn't it be nice if people actually used it properly? Process the documents, normalize them, do the compaction and expansion dance the way the spec intended. That's what Fedify does.
Here's the part that really gets to me, though. Because Fedify actually processes JSON-LD correctly, it's more likely to break when talking to implementations that produce malformed documents. From the end user's perspective, Fedify looks like the fragile one. “Why can't I follow this person?” Well, because their server is emitting garbage JSON-LD that happens to work with implementations that just treat it as a regular JSON blob. Every time I get one of these bug reports, I feel a certain injustice. Like being the only person in the group project who actually read the assignment.
To be fair, there are real practical reasons why most people don't bother with proper JSON-LD processing. Implementing a full processor is genuinely a lot of work. It leans on the entire Linked Data stack, which is bigger than most people expect going in. And the performance cost isn't trivial either. Fedify uses some tricks to keep things fast, and I'll be honest, that code isn't my proudest work.
Anyway, none of this is going anywhere. Just me grumbling into the void. If you're building an ActivityPub implementation, maybe consider using a JSON-LD processor if one's available for your language. And if you're not going to, at least test your output against implementations that do.
#JSONLD #fedidev
@hongminhee I'm reading this thread as a relative noob, but what I see again and again: almost no one "properly" implents #ActivityPub largely because #JSONLD is hard but also because the spec itself is unclear. Most people who get stuff done have to go off-spec to actually ship.
This seems a fundamental weakness of the #fediverse - and that disregarding the limitations coming from base architecture. Seems to pose a mid/long-term existential threat.
What can we do to help improve things?
-
@mariusor@metalhead.club I thought the whole point of signing objects or attaching proofs (none of which I do, mind you) are precisely to save the need to make a new request, which comes with its own overhead.
The good thing is fetching from canonical source will never go out of style.
Aside, it seems like I'm only getting Marius's posts, not silverpills. Makes for an interesting one-sided exchange
-
I have deeply mixed feelings about #ActivityPub's adoption of JSON-LD, as someone who's spent way too long dealing with it while building #Fedify.
Part of me wishes it had never happened. A lot of developers jump into ActivityPub development without really understanding JSON-LD, and honestly, can you blame them? The result is a growing number of implementations producing technically invalid JSON-LD. It works, sort of, because everyone's just pattern-matching against what Mastodon does, but it's not correct. And even developers who do take the time to understand JSON-LD often end up hardcoding their documents anyway, because proper JSON-LD processor libraries simply don't exist for many languages. No safety net, no validation, just vibes and hoping you got the
@contextright. Naturally, mistakes creep in.But then the other part of me thinks: well, we're stuck with JSON-LD now. There's no going back. So wouldn't it be nice if people actually used it properly? Process the documents, normalize them, do the compaction and expansion dance the way the spec intended. That's what Fedify does.
Here's the part that really gets to me, though. Because Fedify actually processes JSON-LD correctly, it's more likely to break when talking to implementations that produce malformed documents. From the end user's perspective, Fedify looks like the fragile one. “Why can't I follow this person?” Well, because their server is emitting garbage JSON-LD that happens to work with implementations that just treat it as a regular JSON blob. Every time I get one of these bug reports, I feel a certain injustice. Like being the only person in the group project who actually read the assignment.
To be fair, there are real practical reasons why most people don't bother with proper JSON-LD processing. Implementing a full processor is genuinely a lot of work. It leans on the entire Linked Data stack, which is bigger than most people expect going in. And the performance cost isn't trivial either. Fedify uses some tricks to keep things fast, and I'll be honest, that code isn't my proudest work.
Anyway, none of this is going anywhere. Just me grumbling into the void. If you're building an ActivityPub implementation, maybe consider using a JSON-LD processor if one's available for your language. And if you're not going to, at least test your output against implementations that do.
#JSONLD #fedidev
@hongminhee from the point of view of someone who is "maintaining" a JSON-LD processing fedi software and has implemented their own JSON-LD processing library (which is, to my knowledge, the fastest in it's programming language), JSON-LD is pure overhead. there is nothing it allows for that can't be done with
1. making fields which take multiple values explicit
2. always using namespaces and letting HTTP compression take care of minimizing the transfer
without JSON-LD, fedi software could use zero-ish-copy deserialization for a majority of their objects (when strings aren't escaped) through tools like serde_json and Cow<str>, or System.Text.Json.JsonDocument. JSON-LD processing effectively mandates a JSON node DOM (in the algorithms standardized, you may be able to get rid of it with Clever Programming)
additionally, due to JSON-LD 1.1 features like @type:@json, you can not even fetch contexts ahead of time of running JSON DOM transformations, meaning all JSON-LD code has to be async (in the languages which has the concept), potentially losing out on significant optimizations that can't be done in coroutines due to various reasons (e.g. C# async methods can't have ref structs, Rust async functions usually require thread safety due to tokio's prevalence, even if they're ran in a single-threaded runtime)
this is after context processing introducing network dependency to the deserialization of data, wasting time and data on non-server cases (e.g. activitypub C2S). sure you can cache individual contexts, but then the context can change underneath you, desynchronizing your cached context and, in the worst case, opening you up to security vulnerabilities
json-ld is not my favorite part of this protocol -
@hongminhee from the point of view of someone who is "maintaining" a JSON-LD processing fedi software and has implemented their own JSON-LD processing library (which is, to my knowledge, the fastest in it's programming language), JSON-LD is pure overhead. there is nothing it allows for that can't be done with
1. making fields which take multiple values explicit
2. always using namespaces and letting HTTP compression take care of minimizing the transfer
without JSON-LD, fedi software could use zero-ish-copy deserialization for a majority of their objects (when strings aren't escaped) through tools like serde_json and Cow<str>, or System.Text.Json.JsonDocument. JSON-LD processing effectively mandates a JSON node DOM (in the algorithms standardized, you may be able to get rid of it with Clever Programming)
additionally, due to JSON-LD 1.1 features like @type:@json, you can not even fetch contexts ahead of time of running JSON DOM transformations, meaning all JSON-LD code has to be async (in the languages which has the concept), potentially losing out on significant optimizations that can't be done in coroutines due to various reasons (e.g. C# async methods can't have ref structs, Rust async functions usually require thread safety due to tokio's prevalence, even if they're ran in a single-threaded runtime)
this is after context processing introducing network dependency to the deserialization of data, wasting time and data on non-server cases (e.g. activitypub C2S). sure you can cache individual contexts, but then the context can change underneath you, desynchronizing your cached context and, in the worst case, opening you up to security vulnerabilities
json-ld is not my favorite part of this protocol@hongminhee take this part with a grain of salt because my benchmarks for it are with dotNetRdf which is the slowest C# implementation i know of (hence my replacement library), but JSON-LD is slower than RSA validation, which is one of the pain points around authorized fetch scalability
wetdry.world/@kopper/114678924693500011 -
reposting so @julian sees this
"I noticed that your inbox endpoint returns 404s (my activities are delivered to personal inbox, not shared)." says @silverpill
-
@hongminhee take this part with a grain of salt because my benchmarks for it are with dotNetRdf which is the slowest C# implementation i know of (hence my replacement library), but JSON-LD is slower than RSA validation, which is one of the pain points around authorized fetch scalability
wetdry.world/@kopper/114678924693500011@kopper @hongminhee I'm glad I'm not the only one who noticed this.
-
@hongminhee take this part with a grain of salt because my benchmarks for it are with dotNetRdf which is the slowest C# implementation i know of (hence my replacement library), but JSON-LD is slower than RSA validation, which is one of the pain points around authorized fetch scalability
wetdry.world/@kopper/114678924693500011@hongminhee if i can give one piece of advice to devs who want to process JSON-LD: dont bother compacting. you already know the schema you output (or you're just passing through what the user gives and it doesn't matter to you), serialize directly to the compacted representation, and only run expansion on incoming data
expansion is the cheapest JSON-LD operation (since all other operations depend on it and run it internally anyhow), and this will get you all the compatibility benefits of JSON-LD with little downsides (beyond more annoying deserialization code, as you have to map the expanded representation to your internal structure which will likely be modeled after the compacted one) -
@julian I don't know as much as I'd like about AT Lexicons. That is, not so much how they work, but what the grand idea is? I don't even understand if Bluesky imagines them being mixed and matched JSON-LD style. I think not?
-
@silverpill @mariusor @hongminhee Fetching from the origin _is_ signature verification. It's just done by the TLS implementation rather than by the ActivityPub implementation.
-
@hongminhee@hollo.social I'll give you my take on this... which is that my understanding of JSON-LD is that with JSON-LD you can have two disparate apps using the same property, like
thread, and avoid namespace collision because one is actuallyhttps://example.org/ns/threadand the other's reallyhttps://foobar.com/ns/thread.Great.
I posit that this is a premature optimization, and one that fails because of inadequate adoption. There are likely documented cases of implementations using the same property, and those concern the actual ActivityStreams vocabulary, and the solution to that is to communicate and work together so that you don't step on each others' toes.
I personally feel that it is a technical solution to a problem that can be completely handled by simply talking to one another... but we're coders, we're famously anti-social yes? mmmmm...
Manu, maker of JSON-LD who also helped with the AP Confs, made this nice video https://www.youtube.com/watch?v=vioCbTo3C-4
JSON-LD is a normative reference to ActivityPub. The context of AP is only 1 line, maybe 4 if you support the official extensions. It does not make anything much larger.
It is for example important if you want to consume the federated wikipedia, wikidata, European Broadcasting Union or these Public Broadcasters https://www.publicmediaalliance.org/public-broadcasters-create-public-spaces-incubator/ but also to know that e.g. mobilizon uses schema.org for addresses.
I give you an example, if you include
"mz": "https://joinmobilizon.org/ns#", "wd": "https://www.wikidata.org/wiki/Special:EntityData/",
"wdt": "https://www.wikidata.org/prop/direct/"in your context, then you know about mobilizon extension but also the whole common knowledge of the world …
I like that, now you can support the whole vocabulary of wikipedia and wikidata which is just JSON-LD.
You get it in all the languages of the world including the properties name.
No problem, if others don't support it, but sad for users. -
Manu, maker of JSON-LD who also helped with the AP Confs, made this nice video https://www.youtube.com/watch?v=vioCbTo3C-4
JSON-LD is a normative reference to ActivityPub. The context of AP is only 1 line, maybe 4 if you support the official extensions. It does not make anything much larger.
It is for example important if you want to consume the federated wikipedia, wikidata, European Broadcasting Union or these Public Broadcasters https://www.publicmediaalliance.org/public-broadcasters-create-public-spaces-incubator/ but also to know that e.g. mobilizon uses schema.org for addresses.
I give you an example, if you include
"mz": "https://joinmobilizon.org/ns#", "wd": "https://www.wikidata.org/wiki/Special:EntityData/",
"wdt": "https://www.wikidata.org/prop/direct/"in your context, then you know about mobilizon extension but also the whole common knowledge of the world …
I like that, now you can support the whole vocabulary of wikipedia and wikidata which is just JSON-LD.
You get it in all the languages of the world including the properties name.
No problem, if others don't support it, but sad for users.PS, I am using the official JSON-LD processor of Manu and contributors, if support in any language is lacking, we just speak to the JSON-LD Group (glad about the 2 webintents coming together now as well ) …
Cause we are social … -
@hongminhee if i can give one piece of advice to devs who want to process JSON-LD: dont bother compacting. you already know the schema you output (or you're just passing through what the user gives and it doesn't matter to you), serialize directly to the compacted representation, and only run expansion on incoming data
expansion is the cheapest JSON-LD operation (since all other operations depend on it and run it internally anyhow), and this will get you all the compatibility benefits of JSON-LD with little downsides (beyond more annoying deserialization code, as you have to map the expanded representation to your internal structure which will likely be modeled after the compacted one)@kopper@not-brain.d.on-t.work @hongminhee@hollo.social expansion is actually really annoying because the resulting JSON has annoyingly similar keys to lookup in a hashmap, wasting cache lines, and CPU time
-
reposting so @julian sees this
"I noticed that your inbox endpoint returns 404s (my activities are delivered to personal inbox, not shared)." says @silverpill
@liaizon@social.wake.st actually, oddly, I did receive @silverpill@mitra.social's response, so it seems to be I can access some replies from Mitra, but not all.
-
@kopper@not-brain.d.on-t.work @hongminhee@hollo.social expansion is actually really annoying because the resulting JSON has annoyingly similar keys to lookup in a hashmap, wasting cache lines, and CPU time
@natty @hongminhee i would imagine a Good hash algorithm wouldn't care about the similarity of the keys, no? -
@hongminhee take this part with a grain of salt because my benchmarks for it are with dotNetRdf which is the slowest C# implementation i know of (hence my replacement library), but JSON-LD is slower than RSA validation, which is one of the pain points around authorized fetch scalability
wetdry.world/@kopper/114678924693500011@hongminhee i put this in a quote but people reading the thread may also be interested: json-ld compaction does not really save that much bandwidth over having all the namespaces explicitly written in property names if you're gzipping (and you are gzipping, right? this is json. make sure your nginx gzip_types includes ld+json and activity+json)
RE: not-brain.d.on-t.work/notes/aihftmbjpxdyb9k7
