Pipeline Architecture: How 22 Scripts, 5 Scanners, and 20 Agents Run an Autonomous Crypto Intelligence System

The EarlyThunder pipeline is not a dashboard that requires manual operation. It is an autonomous system that ingests external data, synthesizes signals, generates prioritized actions, and executes decisions without human intervention in the loop except at defined decision gates. This guide documents the full architecture: component by component, data flow by data flow, and the operational metrics that characterize the system as it currently runs.

The system totals 15,000+ lines of Python across 22 scripts. NASA Power of 10 compliance score: 9.5/10. The 0.5 deduction from perfect reflects a small number of external library calls where internal behavior cannot be fully bounded by P10 rules. Every internal function meets the full standard: single return path, no recursion, bounded loops, explicit error handling, no dynamic memory allocation.

Data ingestion starts at the External API layer. Sources include Binance perpetual and spot APIs (funding rates, OHLCV, order books), DeFi Llama (TVL, protocol revenue, yield pools), CoinGecko (price data, market cap, circulating supply), on-chain indexers for Ethereum, Arbitrum, Sui, and Solana (wallet activity, smart contract events, token transfers), Nansen (smart money wallet labels and flows), and custom web scrapers for protocol documentation and announcement channels. Total external data points ingested per 24-hour cycle: approximately 847,000.

The 6 scanner groups process the raw ingested data at different cadences matched to signal half-life.

Scanner Group 1: Funding Rate Scanner. Cadence: every 8 hours. This is the most frequent scanner because funding rates change at each settlement and opportunities disappear within hours. The scanner covers 124 perpetual pairs, calculates annualized rates, ranks by absolute rate magnitude, filters for minimum open interest ($500K), and outputs the top opportunities to the action queue.

Scanner Group 2: DeFi Yield Scanner. Cadence: every 24 hours. Covers 811 pools. Protocol risk scoring, APY trend analysis (30-day slope), and liquidity depth calculation. Outputs yield opportunities ranked by risk-adjusted return. DeFi yields change daily but not within-day in ways that require more frequent scanning.

Scanner Group 3: Market Intelligence Scanner. Cadence: every 24 hours. Aggregates Fear and Greed Index, stablecoin flow data, sector performance metrics, and BTC dominance trend. Outputs the market regime classification (risk-on/risk-off/neutral) that governs position sizing parameters across all other scanners.

Scanner Group 4: Portfolio Scanner. Cadence: every 24 hours. Monitors the 19-token conviction portfolio against on-chain revenue data. Recalculates P/Revenue ratios as price and revenue data update. Flags tokens where the ratio has moved outside the buy or sell threshold, triggering action queue entries for rebalancing.

Scanner Group 5: Airdrop Scanner. Cadence: every 168 hours (weekly). The airdrop scanner is the lowest frequency because the relevant signals (protocol revenue, user activity, token launch probability) change on weekly or monthly timescales. The 127 tokenless protocols are scored weekly and the top 10 farming opportunities are updated. Less frequent scanning is a deliberate resource allocation choice: airdrop signal processing is computationally intensive and the data does not justify daily refresh.

Scanner Group 6: Testnet Scanner. Cadence: every 24 hours. Monitors testnet activity for protocols approaching mainnet launch. High testnet activity is a leading indicator of imminent mainnet launch, which often precedes airdrop announcements. This scanner identifies new farming targets before they appear on mainstream airdrop tracker sites.

The Synthesizer aggregates outputs from all 6 scanner groups into a unified signal model. It resolves conflicts between scanner outputs (for example, if the Market Scanner classifies the regime as risk-off, the Synthesizer downscales position size recommendations from the Funding Rate Scanner), applies portfolio-level constraints (maximum concentration per token, minimum cash reserve), and produces a prioritized action list.

The Action Queue is the current operational state of the system. It holds 100 pending actions as of the last scan. Actions are classified by priority: P0 (immediate execution, 14 actions) and P1 (execute within 48 hours, 86 actions). Total estimated value across all queued actions: $2.26 million EV, calculated as expected monetary value weighted by probability of success. P0 actions include funding rate position adjustments, yield rebalancing at protocol risk threshold triggers, and two airdrop farming activity requirements with deadlines.

The Executor handles P0 actions automatically within defined parameters. Actions above a capital threshold ($10,000 per single transaction) require a human confirmation gate before execution. Actions below the threshold execute autonomously with full logging. The confirmation gate is the primary point of human-in-the-loop control. P1 actions are presented in a prioritized queue for review at each 24-hour cycle.

The MCP Server is the output interface for external consumption. It sits downstream of the Synthesizer and serves real-time tool calls against the synthesized data. When an AI agent calls get_funding_signals, the MCP server returns the current top opportunities from the Funding Rate Scanner output as processed by the Synthesizer. The server does not re-run the scanner on each call; it serves from the last computed state with freshness metadata.

Monitoring and observability are built into every component. Each scanner run logs input counts, processing time, output counts, and any error conditions. The Synthesizer logs conflict resolution decisions. The Executor logs every action taken with timestamp, instrument, size, price, and outcome. Total log volume per day: approximately 12,000 structured log entries. Log retention: 90 days.

The 20 agents referenced in the title operate at the action execution layer. Each agent is a bounded task runner responsible for a specific action class: funding position management, yield rebalancing, airdrop activity execution, testnet interaction, and so on. Agents operate independently with message-passing interfaces to the central Executor. This architecture allows parallel execution of non-conflicting actions and clean fault isolation when a single agent encounters an error.

Current system status reflects 12 scanners as COMPLETE (all scanner groups operational), 100 actions in the queue, and the MCP server live. The action queue EV of $2.26 million represents the system's current assessment of extractable value from identified opportunities across all five revenue layers. Not all EV will be captured. But the systematic identification and prioritization is the structural advantage over discretionary approaches.

Author: Early Thunder Research Data sources: EarlyThunder internal pipeline metrics, system logs, action queue database, NASA Power of 10 compliance audit output Last updated: 2026-05-21

This content is for informational purposes only and does not constitute financial advice.