Interact

Our project is an automated real world actions platform that lets you seamlessly convert your crypto into real world purchases and services through AI powered agents 🤖. Using blockchain based escrows and smart contracts, we ensure trustless, verifiable payment flows directly on chain 🔗, while offloading the final real world execution to AI agents that act as human proxies 🛠️.

Here’s how it works:

💸 Fiat Off-Ramp Automation: Users fund a secure escrow contract on a blockchain (Hedera, Flare, Flow) 🌐. The system uses a dual attestation workflow:

✅ Users create tasks and deposit funds in native tokens (HBAR, FLR, FLOW). ✅ Agents complete the task (e.g., ordering food, booking flights, shopping Amazon) in the real world using virtual credit cards (VCCs) or payment APIs 💳. ✅ Users verify the completed task or let a time based fallback pay the agent if they don’t verify in time ⏳.

🤖 AI Powered Agents: Our browser based AI agents interpret user tasks, interact with e-commerce sites 🛒, and execute purchases like any human would automating the friction in using crypto for real-world actions 🌍. For example, you can have the agent:

🍕 Order your favorite pizza via Wolt. ✈️ Book flights directly from online travel platforms. 🛍️ Shop for household goods on Amazon.

🔐 Security and Escrow Contracts: We leverage multi chain smart contracts that hold funds in escrow until the agent finishes the task 🛡️. The agent (or user) can claim payment under clear on-chain rules:

⏳ If no agent completes the task within 7 days, the user can cancel for a full refund 💰. 🕒 If the agent finishes but the user doesn’t verify within 3 days, the agent can claim the funds directly. 💸 Direct payouts (no intermediaries) send funds straight to user or agent wallets 💼.

🚀 Onboarding and Flow: ✅ Deposit Funds: Users connect wallets, deposit the required amount in the appropriate blockchain’s token, and lock it in escrow 🔒. ✅ Start AI Agent: Once funds are secured, an AI agent is assigned 🤝. ✅ Agent Completes Task: Via a websocket based real time log view, users can watch their agent executing the task 📺. ✅ Verify and Pay: After completion, users verify the job on-chain and release the escrowed payment to the agent 💸.

🌐 Multi-Chain & Decentralized: The system currently supports Hedera (testnet), Flare, and Flow mainnets each with deployed TaskEscrow contracts 🔗. Users can choose which chain to transact on, opening up flexibility and tapping into different ecosystems 🌀.

📦 Example Use Cases: 🍔 Order food delivery to your home and pay with crypto. 🛫 Book and pay for your next flight entirely with your crypto wallet. 📦 Purchase anything from Amazon using your tokens, automatically fulfilled by the AI agent.

💡 Key Differentiator: We take your crypto from the blockchain to real world goods and services automatically 💥 “automated real world actions that let you turn crypto into real world things.” Unlike traditional fiat off-ramps, our AI agent + escrow system removes the friction of onboarding, compliance hurdles, or third party fiat providers delivering a new paradigm for using crypto 🌐.

This project combines blockchain, AI, and browser automation to create a seamless fiat off-ramp for crypto payments. Here’s the full breakdown:

🔗 Multi-Chain Smart Contracts & Escrow

✅ We wrote a TaskEscrow contract in Solidity that lives on Hedera Testnet, Flare Mainnet, and Flow Mainnet (EVM side).

💰 Users deposit tokens into this contract to create a task.

📦 The contract stores task details (like user, agent, amount, status, timestamps) and securely holds funds in escrow.

🛠️ We leveraged Hardhat for contract deployment, testing, and cross chain verification.

🔒 The dual attestation model (user + agent) ensures payment safety and fairness, with built in timeouts for fallback scenarios.

🧠 AI Agent and Browser Automation

💻 The agent backend is built with FastAPI, using WebSockets (/ws/agent) to provide a live, bidirectional channel for real time updates to the frontend.

🧩 We use LangChain’s ChatOpenAI (langchain_openai.ChatOpenAI) to inject LLM reasoning and dynamic adaptability into each task. The LLM helps interpret, plan, and execute the user’s request, working alongside human like automation flows.

🖥️ Browser automation (via the browser_use library) is orchestrated by the Agent and Controller classes. The agent autonomously fills out forms, clicks buttons, and navigates purchase flows, while sending logs (via an on_step_start hook) back to the user in real-time.

📝 Jinja2 based templates (foodTemplate, flightTemplate) generate context aware task prompts for the LLM, tailored for each type of real world action (ordering food vs. booking flights).

⚡ The entire agent process is asynchronous (using asyncio), ensuring fast, non-blocking updates and scalable concurrency.

🔧 dotenv loads configuration and environment variables, while uvicorn serves the FastAPI app as a lightweight ASGI server.

🚀 The result: an LLM-powered AI agent that turns user prompts into automated real world actions, seamlessly bridging crypto on-chain payments with real world services.

🪄 Frontend Real-Time Logs & Agent Control

🌐 Built with Next.js and React for a responsive, multi page flow (wallet connection, deposit, agent logs, order complete).

🔌 The WebSocket connection streams real time logs from the backend agent to the frontend so users can watch their task progress.

📊 Agent status indicators (running, completed, error) dynamically update the UI.

🎨 We used shadcn/ui for clean, modern UI components, and dynamic theming to match the blockchain selected (e.g., Hedera vs. Flare colors).

💡 Hacky but Effective

💳 To bypass traditional fiat off-ramp restrictions, we let the agent itself hold and spend fiat funds (via VCCs) on behalf of the user, sidestepping direct crypto to fiat conversions. This system reduces a long pain and cause of friction in the ecosystem with utilizing crypto to purchase real world things.

🗃️ The agent is stateless — task data is stored in localStorage on the frontend, and agents pull it live via WebSocket. This avoids writing a persistent backend database, keeping the system lightweight.

🎯 What Partner Tech Helped

📈 Flare’s built in FTSO price feeds let us calculate real time token to USD conversion, making it easy to show fiat values for crypto deposits.

🌐 Hedera & Flow gave us a cheap and fast environment to iterate rapidly on EVM compliant smart contract design.

✨ This project is a true full stack blend of blockchain, real time browser automation, and AI driven actions automating the conversion of crypto to real world services.