Bismillah. Chalo bachon, course mein wapis خوش آمدید.

COURSE: Pakistan Ka Pehla Professional Trading Bot Course

MODULE 3: Market Data Pipeline — API Se Real-Time Data Kaise Fetch Karein

LESSON 3.2: Market Filters — Expiry, Price, Volume Ka Pipeline

Assalam-o-Alaikum! Umeed hai pichla lesson samajh aa gaya hoga. Abhi tak hum ne sirf data fetch karna seekha hai. Lekin API se jo data aata hai, woh kachra aur kaam ki cheezon ka mixture hota hai. Garbage in, garbage out. Agar apne bot ko kachra khilao ge, toh woh trade bhi kachra hi karega.

Aaj ka lesson is course ki rooh (soul) hai. Hum seekhein ge ke hazaron markets mein se sirf woh "golden" candidates kaise nikalne hain jin par hamara bot actually kaam karega. Yeh bilkul waise hai jaise Karachi mein hiring hoti hai: company ke paas 200 CVs aati hain, HR un mein se 20 shortlist karta hai, manager 5 interview karta hai, aur end mein 1 larka select hota hai. Hum apne bot ke liye wohi HR aur manager banenge.

The Pipeline Philosophy: Kachra Saaf Karo, Sona Nikalo

Socho, har filter ek stage hai. Har stage pe kuch markets fail ho kar bahar nikal jaati hain. Jo akhir tak bachti hain, woh hamari high-potential candidates hain.

1. Stage 1: Expiry Filter - Bohot jaldi ya bohot der se expire hone wali markets ko reject karo.
2. Stage 2: Price Filter - Jin markets ki probability hamari strategy ke liye ajeeb hai, unko reject karo.
3. Stage 3: Volume Filter - Jin markets mein paisa hi nahi hai, jahan se exit karna mushkil ho, unko reject karo.

Is approach ko "pipeline" kehte hain. Data ek taraf se enter hota hai, filters se guzarta hai, aur doosri taraf se saaf ho kar nikalta hai. Simple, powerful, aur professional tareeka hai.

Setup: Let's Get Ready to Code

Pehle, kuch helper functions aur dummy data bana lete hain taake code runnable ho. Asal mein, fetch_active_markets seedha Polymarket API se data laayega, lekin yahan hum simulation kar rahe hain.

import datetime
import time
import random

# --- Helper Functions (Yeh asal mein API calls honge) ---

def get_market_price(market):
    """
    Simulates fetching the 'YES' price for a market.
    In a real system, this would be part of the market data object.
    """
    # Polymarket returns prices as floats between 0.0 and 1.0
    return market.get('yes_price', 0.0) * 100 # Convert to percentage

def hours_to_expiry(market):
    """
    Calculates hours until a market expires from its timestamp.
    Polymarket provides expiry timestamps.
    """
    expiry_timestamp = market.get('expiry_ts')
    if not expiry_timestamp:
        return float('inf')
    
    now_ts = time.time()
    seconds_diff = expiry_timestamp - now_ts
    return seconds_diff / 3600

def fetch_active_markets(limit=200):
    """
    MOCK FUNCTION: Simulates fetching a list of active markets.
    In our actual codebase (scanner.py), this would hit the Polymarket API.
    """
    print(f"Fetching {limit} mock markets from the 'API'...")
    markets = []
    now = time.time()
    for i in range(limit):
        markets.append({
            'id': f'market_{i}',
            'question': f'Will event {i} happen?',
            # Expiry from 2 hours to 200 hours from now
            'expiry_ts': now + random.uniform(2 * 3600, 200 * 3600),
            # Price from 1% to 99%
            'yes_price': random.uniform(0.01, 0.99),
            # 24h Volume from $100 to $50,000
            'volume24hr': str(random.uniform(100, 50000))
        })
    print("...Done.")
    return markets

# --- Ab hamara lesson shuru hota hai ---

Yeh setup code copy-paste karlo. Iske neeche hum apne filters likhna shuru karenge.

Filter 1: Expiry Window (The Time Filter)

Hamari theta_sniper strategy (jo hum strategies/theta_sniper.py mein banayenge) time decay pe chalti hai. Matlab hum aisi markets dhoond rahe hain jo jald expire hone wali hain.

• Kyun? Jitna kam time bacha hoga, utna "theta" (time value) tezi se decay hoga, aur hum profit jaldi lock kar sakenge.
• Lekin kitna jald? Agar market 2 ghante mein expire ho rahi hai, toh bohot zyada risk hai. Koi aakhri minute ki news market palat sakti hai. Agar 10 din baad expire ho rahi hai, toh hamara capital bohot der tak phansa rahega.
• The Sweet Spot: Hum 24 se 96 ghante (1-4 din) ka window target karenge. Yeh risk aur reward ka acha balance hai.

Chalo iska filter likhte hain.

def filter_expiring_soon(markets, min_hours=24, max_hours=96):
    """Markets jo 24-96 hours mein expire ho rahe hain."""
    result = []
    print(f"Filtering for expiry between {min_hours}h and {max_hours}h...")
    for m in markets:
        h = hours_to_expiry(m)
        if min_hours <= h <= max_hours:
            # IMPORTANT: Hum market object ko 'decorate' kar rahe hain.
            # Yeh calculated values aagey pipeline mein kaam aayengi.
            m['_hours_to_expiry'] = round(h, 1)
            m['_yes_price'] = get_market_price(m) # Calculate once
            m['_volume_24h'] = float(m.get('volume24hr', 0)) # Clean up volume data
            result.append(m)
    
    # Expiry ke hisaab se sort karna acha practice hai.
    # Is se hum sab se qareeb expire hone wali market ko pehle dekh sakte hain.
    return sorted(result, key=lambda x: x['_hours_to_expiry'])

Code Breakdown:

1. filter_expiring_soon: Yeh function markets ki list leta hai, aur min_hours, max_hours ke parameters.
2. for m in markets: Hum har market pe loop chala rahe hain.
3. h = hours_to_expiry(m): Har market ke liye hum calculate kar rahe hain ke kitne ghante baaki hain.
4. if min_hours <= h <= max_hours: Yeh hai hamara filter condition. Agar expiry hamari range mein hai, toh aage barho.
5. m['_...'] = ...: Yeh part bohot important hai. Hum hours_to_expiry, get_market_price jaise functions ko baar baar call nahi karna chahte. Inefficient hai. Isliye, hum ek baar value calculate karke market dictionary mein hi save kar dete hain. Ise "decorating the object" kehte hain. Maine underscore _ se start kiya hai taake pata chale yeh hamare internal, calculated fields hain.
6. return sorted(...): Hum filtered list ko return kar rahe hain, lekin expiry time ke hisaab se sort karke. Is se execution.py ko aasani hogi ke kis market ko pehle target karna hai.

Filter 2: Price Range (The Probability Filter)

Ab hamare paas woh markets hain jinka time frame theek hai. Agla step hai unki probability check karna. Polymarket mein, price hi probability hoti hai. Agar "Will Pakistan win the next T20?" market ka price 70c hai, iska matlab market 70% chance de rahi hai ke Pakistan jeetega.

Hamari theta_sniper strategy high-probability events pe bet karti hai. Hum aisi markets dhoond rahe hain jinke "YES" hone ka chance bohot zyada ho.

• Kyun? Hum chahte hain ke market hamari direction mein resolve ho. Hum "sure things" pe bet kar rahe hain (ya jitna sure ho sakta hai).
• The Sweet Spot: Hum 60% se 97% price range target karenge.
  • > 60%: Is se neeche risk barh jaata hai. 50-50 wali market mein prediction mushkil hai.
  • < 97%: Is se oopar, profit margin bohot kam reh jaata hai. 99c ki cheez khareed ke 1 dollar pe bechne ka faida nahi, fees mein hi nikal jaayega.

Yeh filter likhna bohot aasan hai, especially list comprehension ke saath.

def filter_price_range(markets, min_pct=60, max_pct=97):
    """
    Filters for markets where the 'YES' price is within a specific range. Market filtering is critical to ensure trades fit your strategy parameters.

---

## 📺 Recommended Videos & Resources
- **[Python List Comprehensions & Filtering](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions)** — Efficient data filtering
  - Type: Python Official Docs
  - Link description: Learn Pythonic ways to filter and transform lists
- **[Time Series Data & Expiry Handling](https://pandas.pydata.org/docs/user_guide/timeseries.html)** — Working with time-based data
  - Type: Pandas Documentation
  - Link description: Learn to handle dates, expirations, and time windows
- **[Market Microstructure Basics](https://www.youtube.com/results?search_query=market+microstructure+trading+price+levels)** — Understanding bid-ask-volume dynamics
  - Type: YouTube
  - Link description: Search "market microstructure price levels volume"
- **[Filtering Techniques for Trading Signals](https://en.wikipedia.org/wiki/Trading_signal)** — Creating effective filters
  - Type: Wikipedia
  - Link description: Learn about signal generation and filtering strategies
- **[Polymarket Market Parameters](https://gamma-api.polymarket.com/docs)** — Actual market field reference
  - Type: API Documentation
  - Link description: Reference available market parameters and their meanings

---

## 🎯 Mini-Challenge
**5-Minute Practical Task:** Create 3 filter functions: (1) filter_by_volume(min_24h_volume), (2) filter_by_expiry(days_until_expiry_max), (3) filter_by_price_range(min_%, max_%). Apply all three filters sequentially to a sample list of 10 markets and verify that only markets matching ALL criteria remain.

---

## 🖼️ Visual Reference

📊 Multi-Stage Market Filtering Pipeline ┌─────────────────────────┐ │ 100+ Raw Markets │ └────────────┬────────────┘ │ ▼ ┌─────────────────────────┐ │ Filter 1: Volume │ │ min_volume >= $3000 │ │ → 45 markets │ └────────────┬────────────┘ │ ▼ ┌─────────────────────────┐ │ Filter 2: Expiry │ │ expires_days <= 30 │ │ → 25 markets │ └────────────┬────────────┘ │ ▼ ┌─────────────────────────┐ │ Filter 3: Price Range │ │ 60% <= price <= 97% │ │ → 8 tradeable markets │ └─────────────────────────┘

---