Bismillah. Chalo bachon, course mein wapis خوش آمدید.

COURSE: Pakistan Ka Pehla Professional Trading Bot Course

MODULE 3: Market Data Pipeline — API Se Real-Time Data Kaise Fetch Karein

LESSON 3.2: Market Filters — Expiry, Price, Volume Ka Pipeline

Assalam-o-Alaikum! Umeed hai pichla lesson samajh aa gaya hoga. Abhi tak hum ne sirf data fetch karna seekha hai. Lekin API se jo data aata hai, woh kachra aur kaam ki cheezon ka mixture hota hai. Garbage in, garbage out. Agar apne bot ko kachra khilao ge, toh woh trade bhi kachra hi karega.

Aaj ka lesson is course ki rooh (soul) hai. Hum seekhein ge ke hazaron markets mein se sirf woh "golden" candidates kaise nikalne hain jin par hamara bot actually kaam karega. Yeh bilkul waise hai jaise Karachi mein hiring hoti hai: company ke paas 200 CVs aati hain, HR un mein se 20 shortlist karta hai, manager 5 interview karta hai, aur end mein 1 larka select hota hai. Hum apne bot ke liye wohi HR aur manager banenge.

The Pipeline Philosophy: Kachra Saaf Karo, Sona Nikalo

Socho, har filter ek stage hai. Har stage pe kuch markets fail ho kar bahar nikal jaati hain. Jo akhir tak bachti hain, woh hamari high-potential candidates hain.

1. Stage 1: Expiry Filter - Bohot jaldi ya bohot der se expire hone wali markets ko reject karo.
2. Stage 2: Price Filter - Jin markets ki probability hamari strategy ke liye ajeeb hai, unko reject karo.
3. Stage 3: Volume Filter - Jin markets mein paisa hi nahi hai, jahan se exit karna mushkil ho, unko reject karo.

Is approach ko "pipeline" kehte hain. Data ek taraf se enter hota hai, filters se guzarta hai, aur doosri taraf se saaf ho kar nikalta hai. Simple, powerful, aur professional tareeka hai.

📊 Multi-Stage Market Filtering Pipeline
┌─────────────────────────┐
│ 200 Raw Markets         │
└────────────┬────────────┘
             │ fetch_active_markets()
             ▼
┌─────────────────────────┐
│ Filter 1: Expiry        │
│ 24h ≤ expiry ≤ 96h     │
│ → ~80 markets           │
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│ Filter 2: Price Range   │
│ 60% ≤ YES price ≤ 97%  │
│ → ~35 markets           │
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│ Filter 3: Volume        │
│ volume24hr ≥ $3,000     │
│ → ~12 markets           │
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│ Golden Candidates       │
│ (Ready for AI analysis) │
└─────────────────────────┘

Setup: Let's Get Ready to Code

Pehle, kuch helper functions aur dummy data bana lete hain taake code runnable ho. Asal mein, fetch_active_markets seedha Polymarket API se data laayega, lekin yahan hum simulation kar rahe hain.

import datetime
import time
import random

# --- Helper Functions (Yeh asal mein API calls honge) ---

def get_market_price(market):
    """
    Simulates fetching the 'YES' price for a market.
    In a real system, this would be part of the market data object.
    """
    # Polymarket returns prices as floats between 0.0 and 1.0
    return market.get('yes_price', 0.0) * 100 # Convert to percentage

def hours_to_expiry(market):
    """
    Calculates hours until a market expires from its timestamp.
    Polymarket provides expiry timestamps.
    """
    expiry_timestamp = market.get('expiry_ts')
    if not expiry_timestamp:
        return float('inf')

    now_ts = time.time()
    seconds_diff = expiry_timestamp - now_ts
    return seconds_diff / 3600

def fetch_active_markets(limit=200):
    """
    MOCK FUNCTION: Simulates fetching a list of active markets.
    In our actual codebase (scanner.py), this would hit the Polymarket API.
    """
    print(f"Fetching {limit} mock markets from the 'API'...")
    markets = []
    now = time.time()
    for i in range(limit):
        markets.append({
            'id': f'market_{i}',
            'question': f'Will event {i} happen?',
            # Expiry from 2 hours to 200 hours from now
            'expiry_ts': now + random.uniform(2 * 3600, 200 * 3600),
            # Price from 1% to 99%
            'yes_price': random.uniform(0.01, 0.99),
            # 24h Volume from $100 to $50,000
            'volume24hr': str(random.uniform(100, 50000))
        })
    print("...Done.")
    return markets

# --- Ab hamara lesson shuru hota hai ---

Yeh setup code copy-paste karlo. Iske neeche hum apne filters likhna shuru karenge.

Filter 1: Expiry Window (The Time Filter)

Hamari theta_sniper strategy (jo hum strategies/theta_sniper.py mein banayenge) time decay pe chalti hai. Matlab hum aisi markets dhoond rahe hain jo jald expire hone wali hain.

• Kyun? Jitna kam time bacha hoga, utna "theta" (time value) tezi se decay hoga, aur hum profit jaldi lock kar sakenge.
• Lekin kitna jald? Agar market 2 ghante mein expire ho rahi hai, toh bohot zyada risk hai. Koi aakhri minute ki news market palat sakti hai. Agar 10 din baad expire ho rahi hai, toh hamara capital bohot der tak phansa rahega.
• The Sweet Spot: Hum 24 se 96 ghante (1-4 din) ka window target karenge. Yeh risk aur reward ka acha balance hai.

Chalo iska filter likhte hain.

def filter_expiring_soon(markets, min_hours=24, max_hours=96):
    """Markets jo 24-96 hours mein expire ho rahe hain."""
    result = []
    print(f"Filtering for expiry between {min_hours}h and {max_hours}h...")
    for m in markets:
        h = hours_to_expiry(m)
        if min_hours <= h <= max_hours:
            # IMPORTANT: Hum market object ko 'decorate' kar rahe hain.
            # Yeh calculated values aagey pipeline mein kaam aayengi.
            m['_hours_to_expiry'] = round(h, 1)
            m['_yes_price'] = get_market_price(m) # Calculate once
            m['_volume_24h'] = float(m.get('volume24hr', 0)) # Clean up volume data
            result.append(m)

    # Expiry ke hisaab se sort karna acha practice hai.
    # Is se hum sab se qareeb expire hone wali market ko pehle dekh sakte hain.
    return sorted(result, key=lambda x: x['_hours_to_expiry'])

Code Breakdown:

1. filter_expiring_soon: Yeh function markets ki list leta hai, aur min_hours, max_hours ke parameters.
2. for m in markets: Hum har market pe loop chala rahe hain.
3. h = hours_to_expiry(m): Har market ke liye hum calculate kar rahe hain ke kitne ghante baaki hain.
4. if min_hours <= h <= max_hours: Yeh hai hamara filter condition. Agar expiry hamari range mein hai, toh aage barho.
5. m['_...'] = ...: Yeh part bohot important hai. Hum hours_to_expiry, get_market_price jaise functions ko baar baar call nahi karna chahte. Inefficient hai. Isliye, hum ek baar value calculate karke market dictionary mein hi save kar dete hain. Ise "decorating the object" kehte hain. Maine underscore _ se start kiya hai taake pata chale yeh hamare internal, calculated fields hain.
6. return sorted(...): Hum filtered list ko return kar rahe hain, lekin expiry time ke hisaab se sort karke. Is se execution.py ko aasani hogi ke kis market ko pehle target karna hai.

Filter 2: Price Range (The Probability Filter)

Ab hamare paas woh markets hain jinka time frame theek hai. Agla step hai unki probability check karna. Polymarket mein, price hi probability hoti hai. Agar "Will Pakistan win the next T20?" market ka price 70c hai, iska matlab market 70% chance de rahi hai ke Pakistan jeetega.

Hamari theta_sniper strategy high-probability events pe bet karti hai. Hum aisi markets dhoond rahe hain jinke "YES" hone ka chance bohot zyada ho.

• Kyun? Hum chahte hain ke market hamari direction mein resolve ho. Hum "sure things" pe bet kar rahe hain (ya jitna sure ho sakta hai).
• The Sweet Spot: Hum 60% se 97% price range target karenge.
  • > 60%: Is se neeche risk barh jaata hai. 50-50 wali market mein prediction mushkil hai.
  • < 97%: Is se oopar, profit margin bohot kam reh jaata hai. 99c ki cheez khareed ke 1 dollar pe bechne ka faida nahi, fees mein hi nikal jaayega.

def filter_price_range(markets, min_pct=60, max_pct=97):
    """
    Filters markets where the YES price is within a specific range.
    Market filtering is critical to ensure trades fit your strategy parameters.
    """
    result = []
    for m in markets:
        price = m.get('_yes_price', get_market_price(m))
        if min_pct <= price <= max_pct:
            result.append(m)
    print(f"Price filter ({min_pct}%-{max_pct}%): {len(result)} markets passed.")
    return result


def filter_volume(markets, min_vol=3000):
    """
    Filters markets with sufficient 24-hour trading volume.
    Low volume = illiquid = slippage kills your profit.
    """
    result = []
    for m in markets:
        vol = float(m.get('_volume_24h', m.get('volume24hr', 0)))
        if vol >= min_vol:
            result.append(m)
    print(f"Volume filter (>=${min_vol:,}): {len(result)} markets passed.")
    return result

Filter Parameters Comparison Table

Start with Balanced settings and tune after 2 weeks of paper trading data.

📺 Recommended Videos & Resources

• Python List Comprehensions & Filtering — Efficient data filtering
  • Type: Python Official Docs
  • Link description: Learn Pythonic ways to filter and transform lists
• Time Series Data & Expiry Handling — Working with time-based data
  • Type: Pandas Documentation
  • Link description: Learn to handle dates, expirations, and time windows
• Market Microstructure Basics — Understanding bid-ask-volume dynamics
  • Type: YouTube
  • Link description: Search "market microstructure price levels volume"
• Filtering Techniques for Trading Signals — Creating effective filters
  • Type: Wikipedia
  • Link description: Learn about signal generation and filtering strategies
• Polymarket Market Parameters — Actual market field reference
  • Type: API Documentation
  • Link description: Reference available market parameters and their meanings

🎯 Mini-Challenge

5-Minute Practical Task: Create 3 filter functions: (1) filter_by_volume(min_24h_volume), (2) filter_by_expiry(days_until_expiry_max), (3) filter_by_price_range(min_%, max_%). Apply all three filters sequentially to a sample list of 10 markets and verify that only markets matching ALL criteria remain.

🇵🇰 Pakistan Case Study: SBP Rate Decision Pipeline

SBP (State Bank of Pakistan) holds Monetary Policy Committee meetings every 6-8 weeks. These create predictable Polymarket markets ("Will SBP cut rates at the next MPC meeting?"). Here is how to tune your pipeline for these markets:

# SBP-specific filter settings
# These markets typically have 5-15 day windows before the MPC meeting
SBP_FILTER_CONFIG = {
    'min_hours': 24 * 3,    # 3 days minimum (avoid last-minute panic)
    'max_hours': 24 * 15,   # 15 days maximum (MPC announcement window)
    'min_pct': 55,          # SBP markets can be closer to 50/50
    'max_pct': 97,
    'min_vol': 10_000       # SBP markets attract serious money
}

# PKR equivalent of $10,000 minimum volume:
# $10,000 × PKR 280 = PKR 28,00,000 (28 lakh)
# This ensures the market has enough depth for your order

def run_sbp_pipeline(markets):
    c = SBP_FILTER_CONFIG
    # Step 1: find SBP markets
    sbp_markets = [m for m in markets if 'sbp' in m.get('question','').lower()
                   or 'state bank' in m.get('question','').lower()]
    # Step 2: apply tuned filters
    stage1 = filter_expiring_soon(sbp_markets, c['min_hours'], c['max_hours'])
    stage2 = filter_price_range(stage1, c['min_pct'], c['max_pct'])
    stage3 = filter_volume(stage2, c['min_vol'])
    return stage3

Why this matters for Pakistanis: You read Dawn and Business Recorder every morning. You know the CPI trends. You know the IMF program constraints. You have context that the average American Polymarket trader does NOT have. Your filter settings can be wider (lower min_pct) because your AI model — combined with your reading — is more accurate on SBP markets than theirs.

💡 Key Takeaways

• A three-stage filter pipeline (expiry → price → volume) reduces 200 raw markets to 10-15 high-quality candidates — this ratio of 90%+ rejection is healthy and intentional
• The 24-96 hour expiry window is the theta sweet spot: enough time certainty for high probability, close enough to expiry for meaningful time decay
• The 60%-97% price range targets markets where the outcome is leaning clearly in one direction but still has meaningful premium left to capture
• $3,000 minimum volume is the liquidity gate — below this, your $50-100 position will move the price against you (slippage)
• SBP and cricket markets warrant adjusted filter parameters because Pakistani traders have genuine contextual edge on these specific market categories