Lorem Computer

This was a fun exploration in bringing together a bunch of new technologies under one interface, while scratching our first itch:

We want there to be an easy way to create archives of web content. What would this look like as a web3 project?

To start to answer this, we knew we could create a web scraper infrastructure and have it receive API commands that it would scrape and store in IPFS. While this is cool in itself, the mapping between a URL–and specifically a Memento URL which is a URL at a timestamp–is one that needs to be easily found into the future. While we haven't totally solved this (Although writing to a blockchain was explored) we think we have a good compromise by using DNSLink mappings and memento timestamps.

We want to make sure the data is stored forever, and so we explored some ways to monetize it. This is where things started to get interesting, because in order to do that you want to interact with other parts of the web3 ecosystem and for this you need an interface. We started to imagine a universal "command line" that could have other commands be added to it. This would allow us to have one implementation of our archiver, but there may be many more. By switching namespaces we'd be able to explore other's contribution to the space.

Once we had an image archive and the corresponding archive retrieval we wanted to monetize this, so we added the Unlock Protocol on the response object. We wanted to go deeper on Filecoin itself and would like to leave this (and usage of libp2p for future work.).

After proving we could create an interface, use it for our initial goal of archiving web content, we started wondering what it would look like to add other APIs. Would they be as easy as one we custom crafted ourselves? It turns out they were. We added NFTPort and quickly made a NFT from our web mementos, and then added Covalent as a more traditional API.

Lorem Computer is a NodeJS CLI tool. Once installed it looks up "commands" via dns queries to find the command in the various user selected "namespaces" that live in IPFS. DNS stores a link to an IPFS "command" file which tells the tool which API endpoint it should connect to.

The user command is then relayed to the API endpoint which may do other work on a remote server, then respond back with text (to STDOUT, STDERR), as well as various "actions" which are things like "open a web page". We imagine we could hook up more of these to tie together parts of the ecosystem. Anyone can add a new endpoint by placing a file on IPFS and running a DNS server.

Our first app on top of Lorem.Computer then is our archival app, that creates screenshot mementos using puppeteer and then storing it in IPFS, and then storing the mapping between IPFS and the URL on our system. This mapping is the key to the system and is stored in DNS in a memento format (so you can search for url grabs of a certain timestamp).

We then built in systems to monetize these interactions by opening a web page and directing the user to a web technology, for example using the Unlock Protocol which can paywall the archived content.

Other apps were easy to create once we'd created these and so we added an NFTPort to create NFTs from the screen mementos, as well as a Covalent API to return ETH prices.

In summary: We use IPFS and DNSLink, (building our own DNS nameserver system), to tie together a Command Line Interface. We scrape web pages, put images up on IPFS, then use the Unlock Protocoll to retrieve them. We also apps built with NFTPort and Covalent to use their API to retrieve information.