How We Built This, Oregon Counselors Directory Engineering
The Scraper Pipeline
The core challenge was acquiring clean, verified data from trusted sources. We built a multi-stage pipeline executing over 15,000 scrapes using custom console JS and BeautifulSoup for solid data extraction, combined with Playwright for JavaScript-heavy network traversal.
Key Metrics
- Up-to-current data via 15,000+ scrapes
- MySQL 8 backend with geolocation-aware search
- Dual-domain deployment sync
OSINT Data Enrichment
Proxy directories intentionally mask provider phone numbers and trap clinic websites behind redirects. We bypass this using a sophisticated Playwright & NLP pipeline to automatically scrape Bing, decode Base64 tracking links, and extract organic text from private practice websites.
Collaboration & Testing
Technology is only as useful as the problem it solves. Instead of building in a silo, we collaborated directly with licensed peers across Oregon, gathering manual feedback and conducting user testing to shape every piece of the directory's workflow and design.
Design Language
Clinical directories are often cold and complex. Oregon Counselors Directory uses a custom HealthTech design system: high-contrast typography paired with a functional 8px grid. Key design decisions:
- OHP Green Engine: Specific color-coding (Sage & Emerald) for Oregon Health Plan providers, making low-cost care easily visible.
- Shadow Profiles: A dual-database approach that maps scraped public data into claimable profiles for provider conversion.
- Adaptive Funnels: Therapists take center stage; experimental treatments are appropriately tiered for high-intent browsing.
Engineering Roadmap
- Stripe Monetization & Onboarding [LIVE]: Multi-tier subscriptions, secure provider dashboard with auto-save, and clean dynamic URL routing.
- Dynamic SEO Sitemap & NLP Tools: Automated tools to generate local search pages, guide content, and city-specialty hub pages.
- Review Engine: A HIPAA-compliant patient feedback loop to verify quality of care across the network.