I build data pipelines that turn raw data into reliable decisions.
3+ years designing production data pipelines across GCP, AWS and Azure, from API ingestion and streaming to warehouse modeling, orchestration and AI-powered automation, with a focus on pipelines that are idempotent, observable and easy to maintain.
I’m Muhammad Mudassir Raza, a Senior Data Engineer who builds and operates end-to-end data pipelines for analytics, marketing and revenue teams. My day-to-day work covers ingesting data from third-party APIs, modeling it in warehouses such as BigQuery and Snowflake, and keeping it accurate with scheduled, self-healing jobs.
I care about the unglamorous details that make pipelines trustworthy: incremental loading, delete-then-append reconciliation, retries with exponential backoff, deduplication and Slack alerting, so a failed run gets noticed immediately and a re-run never creates duplicates.
Beyond batch pipelines, I work with Kafka and PySpark for streaming, Airflow for orchestration, and LLM / n8n automation to turn data into commentary, assistants and workflows. I’m certified across AWS, Databricks and Python, and was recognised in the NASA Astronauts Hackathon.
Professional experience
Where I’ve applied this work.
Production data engineering for marketing and analytics teams, plus real-time streaming and warehouse modeling.
Senior Data Engineer
Marketlytics · Dubai, Remote
Jan 2024 to Present
Led an automated pipeline integrating 17 complex APIs with Azure Functions and Logic Apps, reducing manual reporting effort by 85% and processing time by 70%.
Built an end-to-end SEO analytics pipeline on the Wincher API into BigQuery, with window-function SQL, partitioning, UNNEST for nested JSON and share-of-voice metrics, delivered through Looker dashboards.
Developed a GPT-powered analytics platform for natural language querying on BigQuery with dynamic SQL generation, cutting query resolution time by 90%.
Owned 4 production ETL pipelines (GA4, Google Business Profile, Search Console) on Cloud Functions and Cloud Scheduler: 4× faster reporting, 90% less manual effort.
Designed a Semrush pipeline with incremental loading and dedup (zero duplicates, 40% faster ingestion) and a CallRail pipeline with parallel API calls and retries.
Architected an Outlook email-ingestion framework that standardises scheduled CSV/Excel reports into OneDrive, improving data availability by 3×.
Built a GCP cost monitoring and alerting system with Slack notifications, reducing unexpected cloud spend by 25%.
Built a real-time room occupancy detection system with FastAPI, Kafka, PySpark, PostgreSQL and a Random Forest model, improving accuracy by 20%, with a Flask front end.
Engineered a real-time crypto analytics pipeline on Kafka, AWS S3 and Snowflake, improving data processing efficiency by 30%.
KafkaPySparkFastAPIPostgreSQLAWS S3Snowflake
Data Engineer (Internship)
Karachi Dot Ai · Remote, Pakistan
Jun 2023 to Aug 2023
Managed an Apache Airflow ETL pipeline with Python and MySQL, enabling real-time dashboard updates in Google Sheets.
Led an OLTP to OLAP migration with MySQL and PySpark, establishing a star schema and SCD2 with comprehensive testing.
Built and Dockerised a Flask API for user data submission, improving user experience by 25%.
AirflowMySQLPySparkFlaskDocker
Core capabilities
Skills & technology.
A practical mix of data engineering, cloud infrastructure, automation and AI application development.
Languages & Backend
Python, SQL, Flask, FastAPI, Django
Data Engineering
ETL/ELT pipelines, data warehousing, data modeling, SCD2
Docker, Docker Compose, CI/CD, microservices, Git, Linux
Web Scraping
Requests, BeautifulSoup, Scrapy, Selenium
Visualization
Pandas, Matplotlib, Seaborn, Streamlit
Selected projects
Architecture-led project portfolio.
Each project includes its end-to-end architecture.
Type
Technology
01AI & Automation
Mindify
AI-powered mental wellness companion, Backend Developer, Team DataGrains
An empathetic chatbot that takes a user's text or voice input, detects the underlying emotion, retrieves grounded CBT/psychology insights from a vector database, and uses Gemini 2.0 Flash to craft a caring, practical response, complete with coping tips, next steps, and a cited book insight.
Azure-native CRM data export, Functions, Logic Apps, Microsoft Graph
A scheduled Azure Logic App triggers five separate Azure Functions, each responsible for a slice of Insightly CRM data, quotes, organisations, opportunities, equipment/invoices/users, tasks, and opportunity stages. Every function pulls its entity via the Insightly API, exports it to CSV, then authenticates with Microsoft Graph (MSAL client-credentials flow) to upload the file straight into a shared OneDrive/SharePoint folder, splitting the work across functions keeps each run well under Azure's execution time limits.
Meta Marketing API → BigQuery, multi-endpoint social data sync
Pulls marketing and organic data from Meta's platform through five independent modules, Page insights, post performance, ad performance, ad creative details, and Instagram user insights, each loading into its own BigQuery table. A same-day-run guard checks whether a table already has today's data before pulling again, avoiding wasted API calls when the function gets triggered more than once in a day.
Google Cloud FunctionsPythonMeta Marketing APIInstagram Graph APIBigQuery
Architecture
04Data Pipeline
Nealytics: Multi-Source Revenue Data Pipeline
Cloud Function orchestrating 4 payment platforms into one warehouse
A Google Cloud Function that pulls revenue data from ThriveCart, Stripe (across multiple accounts, fetched in parallel), Shopify, and PayPal, and loads it all into BigQuery, with incremental loading for ThriveCart (only new rows past the last synced date) and parallel fetching for Stripe to keep runtime down. Every run posts a pass/fail summary to Slack, so failures get flagged immediately instead of going unnoticed.
Google Cloud Function, self-healing analytics sync with retry logic
A scheduled Cloud Function that pulls the last 10 days of sessions, events, and key events from a Google Analytics 4 property (by landing page and date), then reconciles it into BigQuery using a delete-then-append pattern, deleting the overlapping date range first so re-runs never create duplicates. Every GA4 call and BigQuery write retries with exponential backoff, and a Slack alert fires the moment a step exhausts its retries.
Google Cloud Function, dual-path GSC sync (by query & by page)
A single Cloud Function that runs two independent sync paths against the same Search Console property: one pulls performance grouped by search query, the other by landing page. The query-level path uses the same delete-then-append pattern as the GA4 pipeline to stay idempotent on re-runs; the page-level path currently appends only. Both land in their own BigQuery table, validated afterwards with clicks/impressions/position roll-up queries.
Google Cloud FunctionsPythonSearch Console APIBigQuerySlack Webhooksfunctions-framework
Architecture
07Data Pipeline
Google Business Profile → BigQuery Pipeline
Multi-location metrics sync, daily + monthly, fetched in parallel
Pulls performance metrics for six Google Business Profile store locations at once, running two sub-pipelines in the same function call: daily metrics for the last 8 days, and monthly search-keyword data for the last month. Each pipeline fetches all six locations in parallel (7 workers), then reconciles into BigQuery with the same delete-then-append pattern used across the other Google-data pipelines, so re-runs stay duplicate-free.
Google Cloud FunctionsPythonGoogle Business Profile APIBigQueryjoblib (parallel fetch)Pandas
Architecture
08Data Pipeline
E-Commerce Data → Snowflake Pipeline (dbt + Airflow)
End-to-end ELT, API extraction, Snowflake warehouse, dbt modeling, Airflow orchestration
Extracts products, users, carts and cart items from the e-commerce platform’s REST API using parallel paginated requests, flattens the nested JSON (addresses, cart line items) into pandas DataFrames, and loads them into Snowflake RAW tables with write_pandas. dbt then builds staging models and analytics-ready marts (dim_products, dim_customers, fct_orders), and an Airflow DAG runs the whole flow daily: extract and load, dbt run, then dbt test.
End-to-end analytics engineering, BigQuery star schema, SCD2, LLM commentary
Pulls live coin data from the CoinGecko API into BigQuery, models it into a proper star schema with a SCD Type 2 table that tracks each coin's market-cap rank history over time, then feeds a daily summary view into two places at once: an LLM that writes newsletter-style market commentary, and a Streamlit dashboard that shows both the charts and that commentary side by side.
PythonCoinGecko APIGoogle BigQuerySQL (window functions)SCD Type 2OpenAI GPT-4o-miniStreamlit + PlotlyLooker Studio
Real-time streaming pipeline with dual query layers
Streams historical stock index data row-by-row through Kafka to simulate a live feed, lands it as JSON in S3, then makes it queryable two different ways: an AWS Glue crawler builds a schema catalog so Athena can query S3 directly with no load step, while Snowflake ingests the same files automatically through Snowpipe for warehouse-side analysis, comparing a serverless query path against a managed warehouse path on the same data.
Incremental sync with parallel fetch and ID-level deduplication
Syncs call tracking data from CallRail into BigQuery without ever re-processing what's already there: it reads the latest timestamp already in BigQuery as a watermark, splits the gap up to today into monthly chunks, fetches all of them in parallel (7 threads via joblib), then diffs the fetched IDs against what's already in BigQuery before uploading, so only genuinely new calls, users, and form submissions ever get written.
Google Cloud FunctionsPythonCallRail APIBigQueryjoblib (parallel fetch)Pandas
Tracks keyword rankings, keyword groups, and competitor positions from Wincher for over a dozen client sites (Michigan Auto Law, IHOP, Outback, Qdoba, Ferguson Roofing, and more), each run as its own scheduled Cloud Function built on one shared Wincher client module. Every run covers a rolling 3-month window split into single-day intervals, fetched in parallel (5 threads) with automatic retry on transient API errors, then truncates and reloads that date range in BigQuery so the numbers stay accurate as Wincher's own data gets revised.
Google Cloud FunctionsPythonWincher APIBigQueryThreadPoolExecutorSlack Webhooks
Architecture
13AI & Automation
Personal Assistant: n8n AI Agent
No-code AI automation, Gemini agent orchestrating 6 Google/web tools
An AI agent built as an n8n workflow: a webhook receives a request, hands it to a Gemini-powered agent that keeps conversational memory across turns, and the agent decides which of six connected tool groups to call, Gmail, web search, Calendar, Docs, Tasks, or Sheets, before the result is sent back through the same webhook.
End-to-end data engineering project, Kafka, S3, Snowflake, fully dockerized
A real-time pipeline that scrapes live crypto prices, streams them through a Kafka producer/consumer pair, cleans and lands them as JSON in an S3 bucket, then auto-ingests into Snowflake via Snowpipe the moment new files land, no manual loading step. Every service (Zookeeper, Kafka broker, producer, consumer) runs as its own Docker container.
A daily-scheduled Airflow DAG that downloads a public NYC taxi dataset, converts it from CSV to Parquet for efficient storage and querying, uploads it to Google Cloud Storage, then loads it straight into BigQuery, each step a separate, retry-able task chained by dependencies. Airflow itself (webserver, scheduler, Postgres metadata DB) runs entirely through Docker.
Recursive multi-state API scrape → Google Sheets → live dashboard
A daily Airflow DAG recurses through every U.S. state to pull consumer financial complaint data from the CFPB's public API, transforms it, and pushes it into Google Sheets, which a Streamlit dashboard then reads directly to give stakeholders a state-by-state, always-current view of complaint trends without anyone touching a spreadsheet by hand.
Apache AirflowPythonCFPB Public APIGoogle Sheets APIStreamlit
Architecture
17Data Pipeline
Scraping 1M+ GitHub Repositories on GCP
Large-scale API harvesting with multithreading + EDA
Paginates through GitHub's public repository listing and fetches over a million repos, enriching each with follower counts, languages, and stargazers through concurrent worker threads with automatic retry on failed requests. The resulting dataset then feeds an exploratory analysis notebook that surfaces trends across languages, audiences, and user attributes, turning raw API output into a dataset someone can actually draw conclusions from.
Data warehouse modeling + Slowly Changing Dimension implementation
Redesigns the classic Northwind OLTP database into a proper star schema: a fact table grained at order/order-detail level with foreign keys to denormalized Employee, Customer, and Product dimensions, plus a Date dimension. MySQL procedures migrate the data from OLTP to OLAP, and PySpark implements a working Slowly Changing Dimension (Type 2) on the Employee dimension, validated against both an insert case and an update case to prove history is tracked correctly.
PySparkMySQLStar Schema ModelingSCD Type 2SQLAlchemy
Architecture
19Data Pipeline
Nested JSON Streaming: Kafka + PySpark
Exam solution, structured streaming, schema flattening, data quality rules
Reads nested Titanic-dataset JSON messages off a Dockerized Kafka topic, applies an explicit schema to work with the nested fields through PySpark's API, then flattens everything to a single level. From there it drops duplicate rows, enforces that key columns are never null, removes fields that aren't needed downstream, fixes numeric types, and writes the cleaned result out as line-delimited JSON.
Retrieval-Augmented Generation chatbot, LangChain, Pinecone, Tavily (based on Eden Marco's LangChain course project)
A documentation assistant, a slimmed-down take on chat.langchain.com, that answers questions about LangChain with cited sources. An ingestion pipeline crawls the docs with Tavily, chunks and embeds them, and indexes them in Pinecone. At question time the app retrieves the most relevant chunks, adds conversational memory so follow-up questions resolve correctly, and has an OpenAI GPT model write the answer inside a Streamlit chat interface.
A custom GPT that takes a business question in plain English, such as “how did sales trend over the last 6 months?”, converts it into SQL, runs that query on BigQuery, and explains the result back in plain English. It lets non-technical teams get sales and analytics answers without writing SQL or waiting on an analyst.
Custom GPTText-to-SQLBigQuerySQLLLM
Architecture
Credentials
Education, certifications & recognition.
Validated training across cloud, lakehouse and data engineering.
2023
Databricks Lakehouse Fundamentals
Databricks
2023
Certified AWS Cloud Practitioner
Coursera
2023
Certified AWS Data Analytics
Udemy
2023
Certified Data Engineer
KarachidotAI
2023
Certified Data Engineer
Emeritus USA
2023
Certified in Python
Google (Coursera)
🎓
Bachelor in Computer Science, GPA 3.59
Federal Urdu University of Arts, Science and Technology, Karachi · 2019 to 2022
🧑🏫
Data Engineering Bootcamp Instructor
DataGrains
🏅
NextGen GPT AI Hackathon
LabLab.ai · 2024
🏅
“Unveiling the Stars”: NASA Astronauts Hackathon
HiCounselor · 2023
Contact
Let’s work together.
Need reliable data pipelines, warehouse modeling or AI-powered analytics? Get in touch.