NFDs on Demeter

Demeter pioneers the use of NFDs (Non-Fungible Data) to establish the first decentralized Big Data & AI Platform dedicated to Agriculture. NFDs ensure data safety, quality, availability, and accessibility, setting a new standard in agricultural data management.

Key Functionalities

1. Create New Data
   • Users can generate new datasets, contributing to Demeter's knowledge base. Data can be shared globally for research, enterprise applications, and humanitarian efforts addressing global hunger issues.
2. Coordination
   • The requested data schema is published on chain through smart contracts (custom and Tablelands). They could later be used for incentivization.
   • Data producer like individual farmers upload their set on lighthouse and register it on chain, aggregating it to the overall dataset
3. Read Data via SQL Queries
   • Advanced users such as data analysts can query Demeter's extensive data library using SQL. Future capabilities may include query scheduling and automated analytics to enhance Demeter's knowledge continuously.
4. Visualize the Data
   • Users can visualize data through graphs and charts, enhancing data interpretation. Future updates will introduce customizable filters and expanded visualization options akin to tools like Tableau and PowerBI.
5. Chat with Demeter About the Data
   • Users of all technical levels can interact with Demeter via a chat interface to inquire about agricultural data. Demeter leverages community-contributed structured data to provide real-time insights.

Implementation Details

• User Verification: Implemented via wallet connect using @web3modal/wagmi for secure user authentication.
• Coordination : the requested data schema is published through a set of smart contract (custom and Tableland's). The data producer like individual farmers can contribute to it by uploading there set on lighthouse and registering it on chain.
• Data Integrity: Ensured by using world IDs via Worldcoin to authenticate users, preventing database pollution by bots. - UI Management: Handled in the repository ui/src/components/world_id.tsx, ensuring a user- friendly interface for interacting with Demeter. - Backend verification: Verification processes are managed within a Firebase function (backend_auth/src/index.ts), maintaining robust security standards. Future Vision

Demeter aims to leverage Web3 capabilities to incentivize data stewards, fostering decentralized data management and enabling DAOs for managing data domains beyond agriculture. This decentralized approach using NFDs ensures high data quality, enhancing AI model training accuracy.

You can view more details about Demeter in this presentation: https://docs.google.com/presentation/d/e/2PACX-1vRpllfHjcvE5Fsw2-VKVeymEroVgDmsjeG18wvk_e0SZ6T3nkzyxDat4r8J_OICQHf5lzpo3ARrJHWv/pub?start=false&loop=false&delayms=3000

A high-level overview of the technical architecture is found at: https://github.com/ppsimatikas/nfd/blob/main/docs/demeter_architecture.png

Technologies Used

1. Filecoin (Lighthouse & Tableland ): Leveraged for decentralized storage of agricultural data, ensuring integrity and availability across the platform.
2. Arbitrum: Implemented smart contract to be used as data catalog.
3. Worldcoin: Utilized for identity verification for the farmers (in this case) to prevent data pollution by bots and ensure the authenticity of contributed data.
4. WalletConnect AppKit: Facilitates seamless connection with Web3 wallets for user authentication and interaction with blockchain components.
5. LangChain: Integrated natural language processing (NLP) capabilities, enabling sophisticated data analysis and interaction through conversational AI.
6. React: Used for building the frontend user interface, enabling dynamic and responsive interactions.
7. Node.js: Powering the backend server for handling API requests and business logic.
8. Firebase: Integrated for authentication and real-time data synchronization, ensuring secure user interactions.
9. Mantine: React component library, layout for the front end easier.

Architecture and Integration Frontend: Developed using React to create a user-friendly interface for data visualization, interaction, and chat functionalities with Demeter.

Backend: Built with Python to handle backend processes, including data fetching, processing SQL queries, and integrating with Filecoin, Arbitrum, and LangChain through Web3.js for blockchain interactions and NLP capabilities.

AuthBackend: Built with Node.js and integrated with WorldCoin to handle the authentication.

Firebase: Firestore is used for real-time data storage and synchronization, ensuring scalability and responsiveness.

Notable Approaches

• Integrating Filecoin with Lighthouse and Tableland for decentralized storage and database management. Leveraging Tableland's ERC721-based decentralized database for efficient metadata storage.
• Utilizes WorldCoin's world ID to verify user (farmers in this case) identities, ensuring that data contributions are authentic and trustworthy. This verification mechanism mitigates the risk of data pollution by bots and unauthorized sources, enhancing the quality and reliability of datasets within the platform.
• Using Arbitrum for a smart contract (data catalog) to improve transaction throughput and reduce gas costs associated with data transactions and smart contract interactions on the platform.