Bismillah. Chalo shuru karte hain.

COURSE: Pakistan Ka Pehla Professional Trading Bot Course

MODULE 3: Market Data Pipeline — API Se Real-Time Data Kaise Fetch Karein

LESSON 3.1: Scanner Module — Gamma API Ka Complete Wrapper

Assalam-o-Alaikum, future bot masters! Pichle lessons mein humne setup aur basic concepts dekhay. Aaj asal kaam shuru hoga. Aapka bot andha hai, usko aankhein deni hain. The Scanner module is the "eyes and ears" of your trading bot, and Gamma API is our free, powerful data source.

Scene simple hai: agar bot ko yehi nahi pata ke market mein chal kya raha hai, tou woh trade kya khaak karega? This lesson is the foundation. Isko aagaya, tou aage sab مکھن (makkhan) hai.

Gamma API - Hamara Data Ka Chashma

So, what is an API? Asaan bhasha mein, yeh ek waiter hai. Aap restaurant (aapka code) mein bethay ho, aapko kitchen (Polymarket ke servers) se data chahiye. Aap waiter (API) ko batate ho, "Bhai, zara saare active markets le aana." Woh jaata hai, kitchen se data laata hai, aur aapki table pe rakh deta hai.

Hum Gamma API use kar rahe hain. Kyun?

1. Free Hai: Koi paisa nahi lagna. Start karne ke liye is se behtar kuch nahi.
2. No Rate Limits (Practically): Baaqi APIs pe aapko har minute mein calls ki limit hoti hai. Yahan tension nahi. Jab tak aap server ko DDoS karne ki koshish nahi kar rahe, you're good. Pakistani developers ke liye ye ek blessing hai.
3. Rich Data: Sirf price nahi, volume, expiry, sab kuch milta hai.

Chalo, ab code pe aate hain. We'll build the core of our scanner.py file today.

Core Functions - Asal Code Shuru

Apne project mein ek file banao, scanner.py. Usmein yeh basic setup daalo:

import requests
import json
from datetime import datetime, timezone, timedelta
import time # For retries

# Gamma API ka base URL. Saari requests yahin se jayengi.
GAMMA_BASE = 'https://gamma-api.polymarket.com'

Ab, ek ek karke humare core functions ko samjhein ge.

1. fetch_active_markets - Market Dhoondna

Bot ko sab se pehle ye pata karna hai ke trade karne ke liye available kya kya hai. Yeh function humare liye Polymarket se saare active markets utha kar layega.

def fetch_active_markets(limit=100):
    """
    Fetches a list of active markets from the Gamma API, sorted by 24hr volume.
    """
    print('[SCANNER] Fetching active markets...')
    try:
        # API ko GET request bhej rahe hain
        r = requests.get(f'{GAMMA_BASE}/markets', params={
            'active': 'true',         # Sirf active markets
            'closed': 'false',        # Jo band ho gaye hain, woh nahi
            'limit': limit,           # Kitne markets fetch karne hain
            'order': 'volume24hr',    # Volume ke hisaab se sort karo
            'ascending': 'false'      # Ziada volume waale pehle
        }, timeout=15) # 15 second ka intezar, warna give up

        # Check karo ke waiter ne sahi order laya hai (status code 200)
        if r.status_code == 200:
            print(f'[SCANNER] Successfully fetched {len(r.json())} markets.')
            return r.json()
        else:
            # Agar koi masla hai, tou error print karo
            print(f'[SCANNER] Error fetching markets. Status: {r.status_code}, Response: {r.text}')
            return []

    except requests.exceptions.RequestException as e:
        # Network ka masla, timeout, etc.
        print(f'[SCANNER] Network error while fetching markets: {e}')
        return []

Code Breakdown:

• requests.get(...): Yeh requests library ka function hai jo API ko call karta hai. Hum usko bata rahe hain ke /markets endpoint pe jao.
• params={...}: Yeh humara order hai. Imagine filtering products on Daraz.
  • 'active': 'true', 'closed': 'false': Hum sirf woh markets maang rahe hain jo abhi live hain.
  • 'order': 'volume24hr', 'ascending': 'false': Iska matlab hai ke jin markets mein pichle 24 ghante mein sabse ziada paisa laga hai, woh list mein sabse upar aayenge. Humara bot high-liquidity markets mein interested hai.
• timeout=15: Bohat important. Agar Polymarket ka server 15 second mein jawab nahi deta, tou humara code aage barh jayega, phansa nahi rahega.
• try...except: Network hamesha reliable nahi hota. Kabhi internet disconnect hojata hai, kabhi API down hoti hai. Yeh block humare code ko crash hone se bachata hai. Agar requests.get fail hota hai, tou except block chalega aur hum ek empty list [] return kar denge.
• r.status_code == 200: HTTP protocol mein, 200 OK ka matlab hai "sab theek hai". Agar 404 (Not Found) ya 500 (Server Error) aaye, tou humein pata chal jayega ke kuch garbar hai.

2. get_market_price - Sahi Daam Nikalna

Ab humare paas markets ki list hai. Har market ka data ek bare se dictionary jaisa hai. Humein usmein se kaam ki cheez, yaani ke "YES" outcome ki price nikaalni hai.

def get_market_price(market: dict) -> float:
    """
    Extracts the YES price (0-100) from a market data dictionary.
    Handles a common API inconsistency where outcomePrices can be a JSON string.
    """
    # 'outcomePrices' key ko access karo. Agar nahi hai, tou default '[0,0]' use karo.
    prices = market.get('outcomePrices', '[0,0]')

    # Common Galti Alert: Kabhi kabhi API yeh string mein bhejta hai
    if isinstance(prices, str):
        try:
            prices = json.loads(prices)
        except json.JSONDecodeError:
            # Agar string ajeeb hai aur parse nahi ho rahi
            print(f"[SCANNER] Warning: Could not parse outcomePrices string: {prices}")
            return 0.0

    # Ensure prices is a list and has at least one element
    if not isinstance(prices, list) or len(prices) == 0:
        return 0.0

    # Index 0 hamesha 'YES' price hoti hai (0.0 to 1.0)
    yes_price_prob = float(prices[0])

    # Probability ko 0-100 ki price mein convert karo aur 2 decimal places tak round karo
    return round(yes_price_prob * 100, 2)

Code Breakdown:

• market.get('outcomePrices', '[0,0]'): Hum market dictionary se 'outcomePrices' ki value nikaal rahe hain. .get() use karne ka faida ye hai ke agar outcomePrices key exist nahi karti, tou code crash nahi hoga, balke yeh humein default value '[0,0]' de dega.
• isinstance(prices, str): Yeh check kar raha hai ke the data type is correct before processing. Scanner modules handle data inconsistencies from APIs gracefully.

Scanner Ka Complete Data Flow

📊 Scanner Data Processing Pipeline
┌──────────────────────────┐
│ Raw API Response         │
│ (JSON from Polymarket)   │
└────────────┬─────────────┘
             │
             ▼
┌──────────────────────────┐
│ Data Validation          │
│ (Check required fields)  │
└────────────┬─────────────┘
             │
             ▼
┌──────────────────────────┐
│ Type Conversion          │
│ (String → List → Float)  │
└────────────┬─────────────┘
             │
             ▼
┌──────────────────────────┐
│ Normalization            │
│ (Scale to 0-100)         │
└────────────┬─────────────┘
             │
             ▼
┌──────────────────────────┐
│ Clean Market Data        │
│ (Ready for strategy)     │
└──────────────────────────┘

API Field Reference Table

📺 Recommended Videos & Resources

• RESTful API Data Parsing in Python — Advanced requests library usage
  • Type: RealPython Tutorial
  • Link description: Learn to handle API responses and edge cases
• JSON Data Structure Best Practices — Working with JSON data
  • Type: Python Official Docs
  • Link description: Learn JSON parsing and validation techniques
• Error Handling in Data Pipelines — Robust data processing
  • Type: YouTube
  • Link description: Search "Python data pipeline error handling"
• API Rate Limiting & Retries — Understanding API constraints
  • Type: Wikipedia
  • Link description: Learn about rate limits and retry strategies
• Polymarket Gamma API Reference — Real API endpoint documentation
  • Type: API Documentation
  • Link description: Reference the actual Polymarket API endpoints

🎯 Mini-Challenge

5-Minute Practical Task: Create a Python function that takes raw market data (with inconsistent outcomePrices formats: sometimes string, sometimes list) and returns normalized price data. Test it with 5 different market data dictionaries and verify it handles all formats correctly without crashing.

🇵🇰 Pakistan Case Study: Building a Karachi-Aware Scanner

Pakistani traders have a language and contextual edge on certain Polymarket categories. Here is how to extend scanner.py to flag these:

# Add to scanner.py
PK_KEYWORDS = [
    'pakistan', 'sbp', 'imran', 'pmln', 'pti',
    'psl', 'pcb', 'babar azam', 'shaheen', 'cpec',
    'rupee', 'pkr', 'imf pakistan'
]

def flag_pakistan_markets(markets: list) -> list:
    """
    Mark markets where a Pakistani trader has cultural context advantage.
    These markets may be mispriced by Western traders who lack local knowledge.
    """
    flagged = []
    for m in markets:
        q = m.get('question', '').lower()
        matched = [kw for kw in PK_KEYWORDS if kw in q]
        if matched:
            m['_pk_edge'] = True
            m['_pk_keywords'] = matched
            flagged.append(m)
    print(f'[SCANNER] Found {len(flagged)} Pakistan-edge markets.')
    return flagged

PKR Volume Context: When you see volume24hr: 50000 in the API response, that is $50,000 USD. At approximately PKR 280/USD, that is PKR 1.4 crore trading in that market every 24 hours. Markets with this level of volume have excellent liquidity for orders up to PKR 50,000 (~$180) with minimal slippage.

💡 Key Takeaways

• The Scanner module is the "eyes" of your bot — without clean, reliable market data, every downstream decision is garbage
• Always use .get() with a default value when reading API response fields — real-world APIs are inconsistent and will crash naive code
• The outcomePrices field from Gamma API can arrive as either a JSON string or a Python list — handle both cases explicitly
• High volume24hr markets are your priority: liquidity means you can enter and exit positions without moving the market against yourself
• Pakistani traders have a measurable cultural and contextual edge on SBP, PSL, PCB, and CPEC-related markets — flag these in your scanner to prioritize them for manual review