100k+
Articles processed per week
Unstructured news, prepared for analysis — not a pile of HTML dumps.
Case Study
Automated data scraping platform powered by AI and LLMs
Rotwand needed one algorithm that could read thousands of news sites in context — not a new scraper per domain. Vstorm shipped a weekly platform on Playwright, LangChain, Pydantic, and LlamaIndex.
The outcome
From hand-built scrapers to a weekly run over thousands of sites
Traditional scraping at Rotwand burned budget and still missed accuracy on unstructured news. The replacement is one information-extraction path that understands context, scheduled every week, and sized to grow with more sources.
100k+ articles a week, scaling toward millions a year, is Rotwand's published throughput — not a STCC clinical suite and not Mixam conversion. The run is Celery Beat; logs sit on Redis.
Thousands
News platforms in the scrape set
One context-aware extractor instead of a bespoke scraper per site.
Weekly
Scheduled run
Celery Beat triggers the pipeline. Redis holds the logs.
About the client
Rotwand is a boutique PR agency in Munich. It treats public relations as a data practice: classic PR plus SEO, with visibility, leads, and numbers the client can check. Coverage has included PRWeek, The Holmes Report, Handelsblatt, ARD, Frankfurter Allgemeine Zeitung, BR, WIRED, and HORIZONT.
The firm is independent and wants a high-tech PR lead. AI is how they keep the monitoring layer from becoming a second agency inside the agency.
Vstorm's impact
Vstorm's impact, the TL;DR
- Weekly automated news gathering — less manual monitoring labour
- Hundreds of thousands of articles processed, with a path to millions a year
- One context-aware extractor over thousands of sources, not a scraper farm
- LLMs for context and sentiment, so media presence is not just a keyword hit
- Playwright, LangChain, Pydantic, LlamaIndex, Celery Beat, Redis
The challenge
Unstructured news, too many sources, scrapers that did not pay
Years earlier Rotwand had started collecting unstructured data with classic scraping. It ate budget, calendar, and engineering time. Accuracy was still not good enough. The question they brought to Vstorm: how to scrape many sources well, without paying for a new parser every time the HTML moved.
The brief was a single algorithm that could read context and pull the right fields from diverse sites — volume a person cannot sit through — using NLP, machine learning, and LLMs, with information extraction as the core, not a pile of CSS selectors.
How we delivered
TriStorm on a monitoring layer, not a one-off scrape
The platform had to run every week, stay legal on copyright, and take more sources without a rewrite.
Retire the scraper farm
Workshops mapped the failed traditional pipeline: cost, time, and accuracy on unstructured news. The replacement had to be one extraction path, not N site-specific jobs.
- Source inventory
- Accuracy gap
- Cost of per-site scrapers
Proof of Value on real news HTML
Playwright for collection. NLP and ML for extraction. LangChain for model operations, Pydantic for validation, Python and cloud for process and store. LlamaIndex for retrieval over messy documents.
- Playwright collectors
- LangChain + Pydantic
- LlamaIndex retrieval
Weekly beat, then scale the corpus
Celery Beat runs the job. Redis holds logs. Copyright and IP constraints stayed in the build. The platform is sized for more sources and for millions of articles a year.
- Celery Beat schedule
- Redis logs
- Copyright-compliant path
How it works
Scrape, extract, validate, retrieve, repeat weekly
Playwright gathers pages from thousands of news platforms. Extraction models read that HTML as language, not as a layout snapshot. LangChain runs the model steps; Pydantic checks the structured output. LlamaIndex speeds retrieval over unstructured leftovers. Celery Beat fires the pipeline each week; Redis keeps the log trail.
Data scraping Playwright across thousands of news platforms
↓
Information extraction NLP and ML prepare the corpus
↓
LLM orchestration LangChain runs models; Pydantic validates
↓
Document retrieval LlamaIndex over unstructured remainder
↓
Weekly run Celery Beat; Redis logs
Rotwand × Vstorm — automated news-scraping pipeline
Results
What Rotwand monitors without a newsroom of scrapers
News gather is automated. Labour on manual monitoring dropped. Throughput is hundreds of thousands of articles, with headroom for millions a year. Weekly updates mean Rotwand can move when a story moves. LLMs add context and sentiment, so a mention is not only a string match. The platform is built to take more sources as the agency grows.
Work with us
Ready to see how agentic AI transforms media-monitoring workflows?
Meet directly with our founders and PhD AI engineers. We will walk through real implementations from 30+ agentic projects and the practical steps to integrate them into your workflows.