casual distributed computing
Lately, I've been contemplating how to make the distributed computing more accessible, more of a thing that a bit tech-savvy person can use of vs something that you only resort to as a larger company. Making it more "casual" in other words.
The back-story of my thinking veering in this direction is that I've noticed that many ideas I’ve been thinking about recently had one thing in common. They were relying on public compute infrastructure
Examples
When I was thinking about allowing people annotate URLs with structured metadata - my primary candidate for backend was Wikidata
In recent thinking on Bringing knowledge and conversation closer together - I've been relying on existing Matrix chat infrastructure
Why do I reach for public compute in my projects?
Some of my projects need backend! I try to go client-side/local-first to minimize complexity when possible, but it not always is!
My projects often have a quality of public good, where it’s less clear that one can easily fund the separate infrastructure. Relying on existing public infra can allow me to bootstrap the project and validate if it's viable/valuable
There are of-course trade-offs. In case of public infrastructure you have
No guarantees of reliability of commercial cloud
No, or minimal ownership/decision-making power
When people think about providing the infrastructure for public good projects - distributed computing often comes up as a potential answer. Most of the personal computing devices out there are idle vast majority of the time and there is a constant temptation to tap into that pool.
But creating a distributed computing system that can produce useful results and not expose you to undue risk is hard.
There is a variety of technical difficulties you face:
orchestration of execution
allocating work and handling failures (nodes will drop out, go offline temporarily mid execution, etc)
ensuring equitable access to computational resources - avoid one client hogging all the resources in the network
Security
you need to trust the compute executing your tasks (or have systems in place to mitigate the risk). The compute nodes
can see your data/code
can return fabricated results
as a compute node you need to ensure that running arbitrary code from the network won't damage your system or steal your data (e.g. by sandboxing the execution)
Heterogeneity of the computing environments
and more
People have successfully solved these issues before, but solutions I've seen have taken trade-offs that leave me unhappy
BOINC solves the trust problems by
having only a small set of vetted organizations admitted to the project
employing techniques like replicating the same job multiple times and taking the majority as the result to minimize probability of invalid results
In case of BOINC people are also not really incentivized to cheat as contributions of compute are voluntary and the only benefit you get out of completing jobs is points on the leaderboard
There is a company that would pool idling GPU capacity from datacenters and allow you to interact with it as one compute cluster
Here only a few big compute providers who are vetted can particitipate
And of-course the companies build internal compute clusters all the time, as they can avoid trust problems by being the owner of all components in the chain.
I was thinking about this recently (partially prompted by being interested in running OpenAI Whisper locally)
And one approach that occurred to me that seems promising is - "What if we scale the problem of distributed computing down?"
What if instead of thinking about how to organize all idle computers in the world into one big network - we tried to solve problem on the level of group of friends.
Your friends are happy to help you (especially if it's easy to do)
which side-steps the problem of "what incentivizes people to join the network"
You can trust your friends
to not steal your data 💬
or send you fake results
You can poke your friends
you can rely on communication and social norms to enforce resource allocation being fair and e.g. one person not hogging all resources
One potential variation on this is running things on a variety of personal compute devices you have
e.g. easily delegating things from a phone to a more powerful machine
Another constraint that seemed important for making the use of distributed computing "casual" is simplicity
There are various solutions that would allow you to set up a compute cluster across a set of machines that you control
Kubernetes
But they usually require technical competency and take a while to set up.
If you were to ask your friends to spend an hour configuring their computer to join your custom distributed computing network, you won't get many takers.
Heterogeneity of compute nodes involved is often an issue as-well
So when thinking about how can things be better - making the setup and joining of the network be really simple - stood out as a core requirement.
With these considerations & constraints in mind - the following architecture came to mind
The experience I'm envisioning is:
You go to a web-page, log in
And your computer is ready to execute or submit tasks
You submit tasks by sending a message to a Matrix chat room
With UI/commands available for common/popular tasks
No setup or technical sophistication required.
Main components this relies on are
Matrix Chat
As a way to relay messages between different compute nodes in the network
Task execution flow
The entry-point web page is a Matrix Chat client.
After you log in - it starts listening for new Matrix events.
A user can submit a task as a Matrix event, that has a Web Worker URL and references to required inputs.
files can be transferred via Matrix's media storage or IPFS
When a client sees such a message - it will pick it up, send an ack message back to the channel and start executing the requested task
(to simplify deciding which node should take the task - one of the nodes in the network can randomly picked to be "coordinator" and would be responsible for allocation decisions)
When the result is ready - the node will post the result back into the channel, or will send it directly to the requester.
Why Matrix as a communication protocol?
The choice of matrix is not essential - any other messaging protocol would suffice, but Matrix is federated, reasonably easy to build on, and I have experience with it.
As with my other Matrix projects this is partially motivated by hope that common messaging layer as a way to solve interoperability
Open source neural TTS that would hopefully allow me to replace Pollycast (https://github.com/Stvad/pollycast/)
Though there are challenges with running this due to memory limits of WASM https://github.com/ggerganov/llama.cpp/issues/97
I'm thinking of building a prototype of this, initially focusing on transcribing voice messages via Whisper. Please reach out if you'd be interested in collaborating on this, have related ideas, suggestions or interesting applications.
Homomorphic encryption made a lot of progress in recent years, but still remains a lot slower then operations over unencrypted data
Sidenotes
you still should be weary, as you can’t ensure the other people’s compute infrastructure is secure from intrusion. So you shouldn’t do anything super sensitive