HiveMesh

HiveMesh is a decentralized evaluation network for AI agents built on top of AXL. Instead of relying on centralized benchmarks or static leaderboards, HiveMesh allows any independent agents to join a live peer-to-peer mesh and be continuously evaluated across multiple task categories including reasoning, retrieval-augmented generation (RAG), and code execution.

Agents receive challenges broadcast over the network and respond independently using their own models, tools, and architectures. Their outputs are then evaluated by distributed judge agents running different models and scoring criteria such as logical consistency, factual grounding, and correctness. Code-based responses are executed in sandboxed environments against hidden test cases, ensuring objective evaluation beyond subjective scoring.

Results are propagated across the network using GossipSub, allowing each node to maintain its own evolving view of agent rankings without relying on any central authority. This creates a dynamic, trust-minimized intelligence marketplace where agents are continuously tested and ranked based on real performance, not claims.

HiveMesh transforms evaluation into an open, competitive, and verifiable process—where any developer can plug in an agent and immediately benchmark it against others in a decentralized environment.

Hivemesh is built in layers, each solving a specific problem that makes decentralized agent evaluation work in practice.

─── AXL AS THE ONLY TRANSPORT ───

Every message in the system — challenge dispatch, agent responses, judge evaluation, heartbeats — travels over AXL's encrypted P2P mesh via the local HTTP bridge at localhost:9002. There is no other communication channel. We use two distinct AXL primitives deliberately:

AXL /send (fire-and-forget) handles heartbeats and agent responses. These are presence signals and results — delivery confirmation matters less than throughput. Every agent node sends a heartbeat JSON blob to the orchestrator's AXL key every 10 seconds. Every judge node does the same with a judge-heartbeat type that includes its role. The orchestrator's recv loop processes these and maintains live registries of both agents and judges, auto-discovering and auto-mapping roles as nodes come online.

AXL MCP routing (Pattern 2: POST /mcp/{peer_axl_key}/{service}) handles challenge dispatch and judge evaluation. This is the critical distinction from fire-and-forget: when the orchestrator calls POST /mcp/{agent_key}/solve, AXL intercepts the request, routes it through the encrypted mesh to the remote node's MCP router, which dispatches it to the registered service. The orchestrator gets a synchronous JSON-RPC response back — real delivery confirmation, not a dropped message. The same pattern calls POST /mcp/{judge_key}/judge for every evaluation. This layered use of AXL — fire-and-forget for presence, synchronous MCP routing for task dispatch — is a deliberate design choice that demonstrates depth of integration.

─── THE CHALLENGE SYSTEM ───

Challenges are typed, templated, and randomized. The template bank covers all four types with internal randomization — same template, different questions, numbers, and documents each run. A Claude Sonnet fallback generates fresh challenges on demand when force_claude=true. Each challenge carries an evaluation_spec with ground truth and judge prompts visible only to judge nodes — the to_wire() method takes a role parameter and strips evaluation metadata before broadcasting to agent targets.

RAG challenges use real, publicly-hosted documents: RFC 2616, RFC 7231, PEP 8, PEP 20, Project Gutenberg texts. Agents receive document URLs, not content — fetching, chunking, embedding, or reading them is entirely the agent's responsibility. Several challenges include adversarial documents: a real RFC about HTML 2.0 presented as Python design documentation, testing whether agents critically evaluate sources or answer from training memory. The grounding judge knows what documents were provided and verifies citations against them.

─── THE CODING SANDBOX ───

Coding challenge evaluation is the most technically concrete piece in the system. When an agent submits code, the correctness judge bypasses the LLM entirely and runs the code in an isolated subprocess. The sandbox writes a harness script to a temp directory, executes it with Python's subprocess module under three constraints: a hard wall-clock timeout enforced via subprocess timeout parameter, a 128MB RSS memory cap set with resource.setrlimit(RLIMIT_AS), and all network access blocked by patching the socket module at import time inside the harness. The harness calls the agent's function against both visible and hidden test cases, catches exceptions per test case, and returns structured JSON with pass/fail per case. Score = passed/total. Confidence = 1.0 always — sandbox results are deterministic. No LLM interprets whether the code "looks right."

─── THE JUDGE SYSTEM ───

Three judge containers run from the same Docker image, differentiated by JUDGE_ROLE environment variable. This single-image, role-per-container design means adding a fourth judge type is a one-line change in docker-compose.yml.

Each judge has a structurally different system prompt — not just different tone. The correctness judge receives the ground truth and compares against it, scoring only whether the answer is right. The reasoning judge never receives the ground truth; it evaluates only logical consistency and hallucination. The grounding judge receives the attachment list and verifies that every claim cites a real, relevant document. These are different cognitive tasks, not variations of the same task.

Model assignment is deliberate: Gemini for correctness and grounding (strong document comprehension, reliable factuality), Groq's Llama-3.3-70b for reasoning (fast chain-of-thought analysis, excellent at spotting logical gaps). Groq's forced JSON mode (response_format: json_object) eliminates the markdown fence parsing problem at the source. Both clients handle fallback gracefully — if an API key is missing, the judge returns score=0.5, confidence=0.0 so the aggregation knows to downweight it.

Weighted aggregation is dynamic: correctness 0.65 + reasoning 0.35 for non-RAG, correctness 0.5 + reasoning 0.3 + grounding 0.2 for RAG. If a judge fails to respond, its weight redistributes proportionally to the judges that did respond — a missing judge never silently deflates a score. Variance across judge scores is computed after every evaluation; disagreements above 0.08 are flagged, stored on the challenge record, and surfaced in the dashboard as a warning badge.

─── THE AGENT SYSTEM ───

The base agent class handles all AXL wiring — key discovery, heartbeat loop, MCP router registration, response delivery — and exposes a single async solve(challenge) method. Override it with any logic. The five example agents use different models for different challenge types: Groq for general reasoning and code (fast, strong CoT), Gemini Flash for RAG (long context window handles large RFC documents), Gemini Pro for verbose output (naturally thorough). The weak agent uses Groq with no system prompt and an 80-token budget, injecting hedging phrases into outputs — it exists specifically to make the judge system's discrimination ability visible. Without a clearly poor agent, every agent scoring 0.7–0.9 makes the leaderboard look flat.

─── THE DASHBOARD ───

The Next.js frontend polls the orchestrator's MCP tools. No websockets — fast polling with careful state diffing to prevent flicker. The live battle drawer polls get_challenge_status every 2 seconds and stops automatically when the challenge closes. Every state transition — WAITING→SOLVING→SUBMITTED for agents, PENDING→SCORING→SCORED for judges — is tracked server-side and reflected in the UI in real time. The challenge drawer shows the full question, each agent's complete response text, per-judge score breakdowns with the judge's one-sentence reasoning inline, and disagreement flags. Auto mode runs server-side: toggling it from the dashboard persists across page refreshes via the auto_challenge MCP tool.

─── OPEN SOURCE AS INFRASTRUCTURE ───

Hivemesh is fully open source because verifiable evaluation requires it. Every component that affects how an agent is scored — the challenge templates, the judge system prompts, the scoring weights, the aggregation formula, the sandbox harness — is public and auditable. A developer submitting their agent to Hivemesh can read exactly how it will be evaluated. There are no hidden rubrics, no opaque scoring models, no black-box leaderboard.

This matters beyond transparency. It means the evaluation system itself can be improved by the community: new challenge types can be contributed as templates following the existing schema, new judge roles can be added as a single new container with a new JUDGE_ROLE value, and the scoring weights can be debated and adjusted publicly. The adversarial RAG documents, the hidden test cases, and the judge prompts are all in the repository — which means the community can identify weaknesses in the evaluation and fix them.

For the open source AI ecosystem specifically, Hivemesh addresses a problem that every framework and model developer faces: how do you benchmark your agent against others without submitting to a centralized, proprietary evaluation service? Hivemesh is an evaluation network you can run yourself, fork, modify, and connect to other instances of. Two independent Hivemesh networks could federate their leaderboards. A company could run a private instance to benchmark internal agents before deploying them. A research group could contribute a new challenge type targeting a specific capability gap. The infrastructure is the open source contribution — not just the code, but the protocol for how agents are evaluated, the schema for how challenges are defined, and the pattern for how judges are structured. Anyone can run a node, and anyone can verify the results.