graphCrawlers

This project is a data aggregation and visualization platform focused on the Web3 ecosystem. It scrapes and indexes content from publishing platforms such as Medium, Mirror.xyz, and other sources where Web3 companies and thought leaders publish technical blogs, announcements, and insights. The scraped data is processed and structured into a Knowledge Graph using Hypergraph, a decentralized graph database. This process enables the transformation of raw content into meaningful, queryable relationships. Once the data is indexed in Hypergraph, it becomes accessible through a GraphQL API, allowing efficient and flexible querying of the information. The frontend UI consumes this API to display the data in an intuitive and interactive format. Also we have try to create a api layer over GRC-20 using arc-20.ts sdk.

Scraper service (Node/Express) Periodically fetches blog posts from sources (Medium). Parses HTML with Cheerio (server-side jQuery) to extract title, author, date, tags, body, og:image, canonical URL, etc. Stores parse HTML in MongoDB for to put that on hypergraph.

Publish to Hypergraph (Next.js + Hypergraph SDK)

We started from the Hypergraph Next.js template. The Next.js app hosts the admin/publishing flows and also uses the Hypergraph SDK to create spaces and publish nodes/edges (articles, authors, projects, tags, relationships).

Publishing is done from frontend only.

GraphQL layer (Hypergraph GraphQL server)

The graph is exposed through Hypergraph’s GraphQL server. The frontend queries this GraphQL server to power lists, detail pages, and graph visualizations.

Frontend UI (Next.js)

Next.js app consumes GraphQL endpoints (Hypergraph) and displays content