As distributed protocols like IPFS become more ubiquitous we will start to see many new applications develop on the distributed web with a focus primarily on offline first ,privacy and reduced bandwidth.
tldr; GitHub project at bottom of article ↓
This article cover an AWS Lambda copy cat application on IPFS. We will be using Golang, Docker, IPFS and Python. One thing to note, is that this project will not be using any Blockchain.
It’s important to see how new tech like IPFS is not directly tied to Blockchain. We can build useful applications with just IPFS and existing time tested tools.
Lambda-like on IPFS
IPFS PubSub Compute is an early stage IPFS powered dapp. It aims to empower functions as a service similar to AWS Lambda. I plan to continue to work on this app as it has a-lot of potential for building portable micro services.
The app allows a user to execute Python 3.7 code in a docker container when triggered by a IPFS PubSub message.
Using IPFS PubSub as the messaging backbone allows us to separate the code writer and the code executor. While there are many mature message queueing systems available — we want to build around IPFS because of its offline nature and promising project development. We also want to use IPFS (not just the PubSub) since we need a way to distribute the code packages we develop. IPFS is a perfect way to share.
Introducing — IPFS PubSub Compute
IPFS PubSub Compute (for lack of a better name at the time) has a very straight forward execution flow.
First we need to write the code and the event — just like in AWS Lambda.
Here we are using some code that corresponds to a solution to the second project Euler problem (its about adding fibonacci numbers)
Sending code and event to IPFS PubSub Compute server
We send the code and the event acting on the code to our local server running at localhost:8769 this just takes out input and relays it on to a PubSub channel. The actual code is shown below.
code := c.Query("code")event := c.Query("event")fmt.Println("Publishing")msg := `{"action": "execute","data": {"event": "` + event + `","code": "` + code + `"}}`
Now that the message is fired off into the IPFS PubSub channel (we use test as our default channel)
Data sent to executor node over IPFS PubSub
Another IPFS node (or the same one for the demo) is listening to the channel and executes a docker run when an “execute” message is broadcasted on the test channel. The actual code shown below.
cmd := "docker"args := []string{"run","--rm","-v","THE_CODE_LOCATION:/var/task","lambci/lambda:python3.7","lambda_function.lambda_handler",event,}out, err := exec.Command(cmd, args...).Output()
Now the code is copied from the message to a temporary location. The docker container then reads the code executes it with the given event input.
Docker container processes code and sends back to server
The docker container is an exact copy of AWS’s Python 3.7 Lambda container and is used as a reference for local development. In our case it works perfectly as a contained Python worker. Since we can create and destroy containers quickly, attaching these container to the network with an event handler we have created a tiny IPFS power distributed computer! With some effort this system could be scaled across many machines and make use of all of the benefits of IPFS.
An immediate feature want 🙋
One major improvement — and next step is to allow users to deploy their code to IPFS as an object. This way we can refer to the code as a single CID and distribute the function to others. If they also have ipfs-pubsub-compute, they’d be able to execute the code themselves. With a few additions to the code the application could handle (CID, event) requests and just fetch the code on execution.
These efforts almost achieved an offline first server-less architecture, but for sure need a lot of work to be production ready.
A demonstration
Based on the concepts above and a running go-ipfs node we can build and run the app! 🙌
Edit to executions demo
Above is a working example of IPFS PubSub Compute, where a simple lambda function is running the helper fibonacci function — and returning the sum of all the even fibonacci numbers. Then the function is edited to show a list of the numbers and not the sum. As you can see both times the output is printed in the lower pane.
This demonstrates the end to end process of
- A user editing the code in the UI
- The code is base64 encoded and sent to the server along with the event (click the 🗒️ symbol next to the execute button to open that pane)
- The server sends the encoded value over IPFS PubSub
- The IPFS PubSub listener decodes the message and copies the it into a Docker container
- Docker container executes code (exact replicate of the AWS Lambda Python 3.7 environment)
- The lambda function’s results are forwarded back to the server
- The server shows the results on screen
Check out the project here: ipfs-pubsub-compute
Next Steps
- Allow IPFS code uploads
- Server can fetch code
- Export to AWS compatible ZIP file
- Allow user to import custom libraries
- Add Deploy to IPFS button
The whole application is less than 130 lines of go code, and it require docker, go and the lambci/lambda:python3.7 docker image. It is a boilerplate for simple apps that interact with IPFS’s API. Here we only use the PubSub API’s but IPFS has a rich storage API as well.
There are many ways this small project could go, but hopefully it serves as a simple — but useful app on IPFS. No Blockchain, we still use an HTTP server and a copy of a commercial Docker container. Web 3.0 isn’t going to be about replacing existing services —but about connecting and leveraging them.
Thanks for getting through this adventure with me. Please show support and 👏 (clap) for this article.
Remember you can clap up to 50 times, and clapping for this article would be really helpful in the Medium algorithm.
Any support is greatly appreciated 💝
Also, please answer the questions and add to the discussion below