How We Built This — Oregon Counselors Directory Engineering
The Scraper Pipeline
The core challenge was acquiring clean, verified data from trusted sources. We built a multi-stage pipeline executing over 15,000 scrapes using custom console JS and BeautifulSoup for robust data extraction, combined with Playwright for JavaScript-heavy network traversal.
Key Metrics
- Up-to-current data via 15,000+ scrapes
- MySQL 8 backend with geolocation-aware search
- Dual-domain deployment sync
OSINT Data Enrichment
Proxy directories intentionally mask provider phone numbers and trap clinic websites behind redirects. We bypass this using a sophisticated Playwright & NLP pipeline to automatically scrape Bing, decode Base64 tracking links, and extract organic text from private practice websites.
Collaboration & Testing
Technology is only as useful as the problem it solves. Instead of building in a silo, we collaborated directly with licensed peers across Oregon — gathering manual feedback and conducting user testing to shape every piece of the directory's workflow and design.
Design Language
Clinical directories are often cold and complex. Oregon Counselors Directory uses a custom HealthTech design system: high-contrast typography paired with a functional 8px grid. Key design decisions:
- OHP Green Engine: Specific color-coding (Sage & Emerald) for Oregon Health Plan providers, making low-cost care easily visible.
- Shadow Profiles: A dual-database approach that maps scraped public data into claimable profiles for provider conversion.
- Adaptive Funnels: Therapists take center stage; experimental treatments are appropriately tiered for high-intent browsing.
Engineering Roadmap
- Stripe Monetization & Onboarding [LIVE] — Multi-tier subscriptions, secure provider dashboard with auto-save, and clean dynamic URL routing.
- Dynamic SEO Sitemap & NLP Tools — Automated tools to generate local search pages, guide content, and city-specialty hub pages.
- Review Engine — A HIPAA-compliant patient feedback loop to verify quality of care across the network.