#pandas · 8sync News

How to Find Stock-Specific Moves in the S&P 500 with Python

A step-by-step guide to building a Python scanner that identifies stock-specific (idiosyncratic) moves in the S&P 500 by separating market-driven returns from stock-specific residuals. The approach uses rolling OLS regression via statsmodels to estimate each stock's beta and alpha against SPY over a 60-day window, computes drift-corrected z-scores on residuals, confirms signals with volume ratios, and checks news headlines via the EODHD API. Multi-day compounded residuals (1, 3, 5-day windows) help distinguish one-day spikes from sustained moves. Real examples from June 12, 2026 illustrate the scanner in action, including SATS (-11%, 6x volume), LEN (z-score -3.9), and a TKO ticker collision edge case. Visualizations include actual vs. expected return scatter plots, z-score bar charts with volume labels, and a trailing abnormal returns heatmap.

I Spent an Hour on a Data Preprocessing Task Before Asking Gemini

A data scientist shares a hands-on experience spending an hour manually solving a Pandas data preprocessing task — extracting a probability value by matching a category ID across list columns — before asking Gemini to do the same. Gemini solved it in seconds, but its first solution used the non-vectorized apply function. Only because the author understood Pandas well enough did they recognize the suboptimal approach and prompt Gemini for a vectorized alternative using explode and regex. The takeaway: LLMs dramatically boost productivity, but domain knowledge is still essential to evaluate and improve AI-generated code.

Retrieval Is Filtering, Not Search: A Mental Model for Enterprise RAG

A mental model for enterprise RAG that reframes retrieval as a filtering problem on two structured DataFrames — line_df (document text) and toc_df (table of contents) — rather than a vector similarity search. The core insight is separating anchor scope (small, precise: where the signal is detected) from context scope (large, sufficient: what gets passed to the LLM for generation). The article walks through four question types that require different retrieval strategies, explains why cosine top-k fails on most of them, and introduces three context expansion strategies: paragraph, section, and window expansion. It also covers the anchor/context distinction, how the two DataFrames collaborate via section_id joins, and when to use LLM calls versus deterministic rules for section boundary detection.