RAW NEWS (4 sources)
    │
    ▼
┌─────────────────────────┐
│  INGEST                  │  ← aiohttp + feedparser
│  80-120 articles/hour    │    (parallel fetching)
└──────────┬──────────────┘
           │
           ▼
┌─────────────────────────┐
│  DEDUPLICATE             │  ← difflib.SequenceMatcher
│  Remove cross-source     │    (similarity > 0.8 = duplicate)
│  duplicates              │
│  80 articles → ~45       │
└──────────┬──────────────┘
           │
           ▼
┌─────────────────────────┐
│  STAGE 1: FILTER         │  ← Gemini Flash (cheap)
│  "Is this relevant to    │    Cost: ~$0.0001 per article
│   any tracked market?"   │
│  45 articles → ~5-8      │
└──────────┬──────────────┘
           │
           ▼
┌─────────────────────────┐
│  STAGE 2: SCORE          │  ← Claude Sonnet (deep reasoning)
│  "How does this change   │    Cost: ~$0.005 per article
│   the implied prob?"     │
│  Output: trading signal  │
└──────────┬──────────────┘
           │
           ▼
┌─────────────────────────┐
│  SIGNAL OUTPUT           │  → Feeds into execution engine
│  market_id, new_prob,    │    (Module 4)
│  confidence, urgency     │
└─────────────────────────┘

COST AT SCALE:
Without 2-stage: 45 articles × $0.005 = $0.225/hour = $5.40/day
With 2-stage: 45 × $0.0001 + 6 × $0.005 = $0.0345/hour = $0.83/day
SAVINGS: 85% ← This is why the pipeline architecture matters

SOURCE PRIORITY TABLE:

| Source | Speed | Credibility | Best For | RSS Available |
|--------|-------|------------|----------|--------------|
| Twitter/X | Real-time | Variable | Breaking news, leaks | API/Nitter |
| Dawn | 1-2 hours | High | Pak politics, economy | Yes |
| Business Recorder | 1-3 hours | High | SBP, finance, trade | Yes |
| Geo News | 30 min | Medium-High | Breaking Pakistan news | Yes |
| Reuters | 2-4 hours | Highest | Global events | Yes |
| AP News | 2-4 hours | Highest | US politics, global | Yes |

KEY INSIGHT:
Pakistani sources (Dawn, Geo, BR) provide 2-6 HOUR information lead
over Western wire services for South Asian events.

If you're trading a Pakistan-related market and you only read Reuters,
you're the LAST person to know. Pakistani news sources are your edge.

import asyncio
import aiohttp
import feedparser
from datetime import datetime, timedelta

class NewsIngester:
    """Fetches headlines from multiple RSS sources concurrently."""

    SOURCES = {
        "reuters": "https://feeds.reuters.com/reuters/worldNews",
        "ap": "https://rsshub.app/apnews/topics/world-news",
        "dawn": "https://www.dawn.com/feeds/home",
        "business_recorder": "https://www.brecorder.com/feeds",
        "geo": "https://www.geo.tv/rss/1/0",
    }

    async def fetch_feed(self, session, name, url):
        """Fetch and parse a single RSS feed."""
        try:
            async with session.get(url, timeout=10) as response:
                text = await response.text()
                feed = feedparser.parse(text)
                articles = []
                for entry in feed.entries[:20]:  # Last 20 per source
                    articles.append({
                        "source": name,
                        "title": entry.get("title", ""),
                        "summary": entry.get("summary", "")[:500],
                        "url": entry.get("link", ""),
                        "published": entry.get("published", ""),
                        "fetched_at": datetime.utcnow().isoformat()
                    })
                return articles
        except Exception as e:
            print(f"[WARN] {name} fetch failed: {e}")
            return []

    async def fetch_all(self):
        """Fetch all sources in parallel — 2-4 sec vs 10-15 sec sequential."""
        async with aiohttp.ClientSession() as session:
            tasks = [
                self.fetch_feed(session, name, url)
                for name, url in self.SOURCES.items()
            ]
            results = await asyncio.gather(*tasks)
            # Flatten list of lists
            all_articles = [a for batch in results for a in batch]
            print(f"[INGEST] Fetched {len(all_articles)} articles from {len(self.SOURCES)} sources")
            return all_articles

from difflib import SequenceMatcher

def deduplicate(articles, threshold=0.8):
    """Remove near-duplicate articles based on title similarity."""
    unique = []
    for article in articles:
        is_dupe = False
        for existing in unique:
            similarity = SequenceMatcher(
                None,
                article["title"].lower(),
                existing["title"].lower()
            ).ratio()
            if similarity > threshold:
                is_dupe = True
                # Keep the one from the higher-credibility source
                break
        if not is_dupe:
            unique.append(article)

    print(f"[DEDUP] {len(articles)} → {len(unique)} unique articles")
    return unique

STAGE_1_PROMPT = """You are a news relevance filter for a prediction market trading bot.

Active markets being tracked:
{market_list}

For each headline below, respond with ONLY the relevant headlines
(ones that could affect the probability of any tracked market).
Return one headline per line. If none are relevant, return "NONE".

Headlines:
{headlines}"""

# Example market_list:
# - "Will SBP cut rates below 15% by Dec 2026?"
# - "Will India-Pakistan bilateral trade resume by Q3 2026?"
# - "Will PIA complete privatization by June 2026?"

STAGE_2_PROMPT = """You are an expert prediction market analyst.

Market question: "{market_question}"
Current market price (implied probability): {current_price}
Resolution criteria: "{resolution_criteria}"

Breaking news headline: "{headline}"
Summary: "{summary}"
Source: {source}
Published: {published}

Based on this news, analyze:
1. How does this headline change the implied probability?
2. What is your confidence in this assessment?
3. How urgent is this signal (should the bot act now or wait)?

Return JSON only:
{{
    "new_probability": 0.XX,
    "confidence": 0.XX,
    "urgency": "high" | "medium" | "low",
    "reasoning": "One sentence explanation",
    "direction": "bullish" | "bearish" | "neutral"
}}"""

DAILY AI COSTS:

                    WITHOUT 2-STAGE     WITH 2-STAGE
Articles processed   1,080/day            1,080/day
Stage 1 (Flash)     —                    $0.10/day
Stage 2 (Sonnet)    $5.40/day            $0.72/day (only 144 articles)
─────────────────────────────────────────────────────
TOTAL               $5.40/day            $0.82/day
MONTHLY             $162/month           $24.60/month
SAVINGS             —                    85%

At 1 PKR = $0.0036:
Without: PKR 45,000/month → TOO EXPENSIVE for most traders
With:    PKR 6,800/month  → AFFORDABLE even for students

MUST-FOLLOW TWITTER ACCOUNTS (by category):

ECONOMY/FINANCE:
├── @StateBank_Pak — SBP rate announcements (CRITICAL for rate markets)
├── @FinaborPk — Pakistan finance news
├── @baborjakhar — Business journalist, breaks SBP news
└── @BusinessRecrdr — Business Recorder breaking news

POLITICS:
├── @PakPMO — Prime Minister's Office
├── @ForeignOfficePk — Foreign ministry (bilateral trade, BRICS)
├── @NAaborPakistan — National Assembly proceedings
└── @dawn_com — Dawn breaking news

SPORTS:
├── @GeoSuper — Cricket, PSL (for cricket market trades)
├── @TheRealPCB — Pakistan Cricket Board
└── @ESPNcricinfo — International cricket updates

INTERNATIONAL:
├── @IMFNews — IMF program updates (critical for PKR markets)
├── @Reuters — Global breaking news
└── @AP — Associated Press

OPTION 1: Twitter API v2 ($100/month)
├── Real-time filtered stream
├── Push-based (instant delivery)
├── Best for serious trading bots
└── Rate limit: 500,000 tweets/month

OPTION 2: Nitter RSS Bridge (FREE)
├── nitter.net/[username]/rss
├── 5-10 minute delay vs real-time
├── No API key needed
├── Fragile (Nitter may go down)
└── Good enough for learning/testing

OPTION 3: RSS.app or RSSHub ($0-5/month)
├── Converts Twitter profiles to RSS
├── 1-5 minute delay
├── More reliable than Nitter
└── Recommended starting point

{
    "market_id": "will-sbp-cut-rates-q2-2026",
    "headline": "State Bank signals dovish shift in latest quarterly review",
    "current_price": 0.42,
    "new_probability": 0.65,
    "confidence": 0.78,
    "urgency": "high",
    "direction": "bullish",
    "source": "dawn",
    "timestamp": "2026-03-26T09:15:00Z",
    "reasoning": "Quarterly review language shifted from 'maintaining' to 'reviewing appropriate levels' - historically precedes rate cuts"
}

TRADING RULES (configure in your bot):

IF new_probability − current_price > 0.15
   AND confidence > 0.70
   AND urgency == "high"
   → EXECUTE BUY (limit order at current_price + 0.02)

IF new_probability − current_price > 0.10
   AND confidence > 0.60
   AND urgency == "medium"
   → QUEUE for next batch execution (every 15 min)

IF confidence < 0.50
   OR abs(new_probability − current_price) < 0.05
   → SKIP (signal too weak)

IF urgency == "high" AND direction == "bearish"
   AND you hold YES shares
   → ALERT: consider selling current position