ChainChat

This project aims to provide a multimodal interface for managing a personal finance portfolio that lives on the blockchain. Personal finance and money management is tough as it is, and adding blockchain complexity does not make it any better! By using text and audio interfaces with LLM models, the user interfaces for complicated financial products can be simplified. On the backend, this agent will be trained to handle financial inquiries, providing a lay-user with education on financial literacy and blockchain related knowledge. It may recommend portfolios and handle blockchain transactions to manage user assets.

In the backend, this project uses Coinbase Agentkit and OpenAI's GPT-4o realtime model for audio interpretation. Thanks to Coinbase's Agentkit and CDK integration, the user can interface with the base-sepolia network for cheap and fast transactions. In the frontend, we have a Next.js app running with Privy integration for easy wallet creation. We also ran a Gaia node in the backend as an alternative to using OpenAI's API, but those local nodes were not powerful enough to manage the agentic tasks. My coordination between the Coinbase Agent and GPT-4o realtime is hacky. Since I could not get tools to work with GPT-4o (it may not be supported), I was running two agents simultaneous in the background. The User would speak to the 4o agent, while the CDP agent would listen in for blockchain related context, and perform actions when it detected that. Unfortunately, I could not get a feedback loop going between the two agents, so the the realtime agent is not context aware of what the CDP agent replies.