About This Site
Hello world! RealmCharts is a project for one of my favorite online bullet hell multiplayer games RotMG. I've always wanted to learn more about the intersection of web development and data analysis, and this project was the perfect combination of the two. All the code is open-sourced on GitHub.
RealmCharts provides a live dashboard for insights on RotMG player count and Steam review analysis. Enjoy!
What We Show
- Players Now: Minute-level granularity of unique players seen in the last 60 minutes (excluding <2 star players)
- 24-hour Forecast: High accuracy time series forecast for 1-, 6-, 12-, and 24-hour horizons
- Steam Analysis: Review volume by day and %positive reviews over your time period of choice
- RotMG Milestones: Annotations for large changes in game mechanics and player review responses
How We Model
Every statistic on RealmCharts is created through multi-step ETL pipelines on live data. Here is the general framework:
- Data Ingestion: player count data is sampled every minute and filtered for anomalies and game maintenance periods; steam reviews are sampled every hour. Data is aligned to UTC and player count data is upsampled to five-minute averages for forecasting.
- Forecasting: we use the powerful Unobserved Components time series model to capture multiple consistent patterns in RotMG player behavior, leading to 97%+ MAPE (≈70-80 MAE) for one hour forecasts.
- Live statistics: Highcharts and pure JavaScript provides an on-demand experience at the minute-granularity.
Forecasting Methodology
RotMG player data contains multiple patterns we used to obtain high accuracy forecasts. Cyclical effects (called seasonalities) were modeled carefully: there is a daily seasonality and a weekly seasonality (more players playing on weekends). If you look closely, each days' peak player count contains two smaller peaks: the first one for peak EU time and the second for peak US time. If you look even more closely, the EU peak is stronger than the US peak on Friday-Sunday while the US peak is stronger than EU during Monday-Thursday, gradually weakening until Thursday where EU/US peaks are about the same. Multiple seasonal components were utilized to model the various cyclic effects and double EU/US peak.
Maintenances with significant content (MotMG) and season starts cause shock effects: more players return to the game than prior. Cubic spline basis functions were used to propogate the shock effect (after Tuesday maintenance) from the beginning of the week until the end of the weekend, allowing interpretations on the direct additive effect of the shock.
Lastly, the general trend of players increases after a significant maintenance/season start, and slowly decreases week by week until the next maintenance. This was handled with a smooth trend parameterization in Unobserved Components.
I'll go much more into detail on the modeling philosophy, progression, and automation over time in a separate blog post. Stay tuned!
Attribution and Technologies
This website uses
Also used are
Changelog
10/7/2025
- Documented API for public use
- Updated About page with details on the backend process
10/5/2025
- Modeled new season/big event shock effect with cubic spline basis, increasing forecast accuracies and fitting the data generation process better
9/28/2025
- Added forecast performance last 48 hours
- Added significant event/content release annotations in Steam review chart
9/26/2025
- Upgraded forecaster to use weekly seasonality in estimating day-to-day effects, and daily seasonality to estimate the within day EU/US peak
9/25/2025
- Refactored large database codebase using factory method and decorators for exception handling
9/24/2025
- Analyzed model and forecast stability using condition number/eigenvalues of the state transition matrix, ACF/PACF plots, standardized forecast errors
- 2 weekly harmonics + AR error component 3 on three weeks and three days of data was extremely unstable and forecasts had larger than acceptable prediction intervals
- Began saving 1/6/12/24h forecasts for forecast accuracy comparison
9/23/2025
- Optimized page loading speed from 5 seconds to below 2 seconds with Gunicorn async workers, batching JS async tasks in
Promise.all, `gzip` compression, caching long review query
9/23/2025
- Forecast now predicts 0 until DECA's estimated end of maintenance time
- Disabled forecast when data is bugged (enabled 60 minutes after data collection returns)
- Few mobile UI optimizations (scroll with one finger, fixed navbar overflow)
9/21/2025
- Deployed automated ML pipeline deployment for forecasting player counts using unobserved components
9/20/2025
- Added ETL pipelines from dirty data -> forecast ready: cleaned sequential unreliable playercounts + 60 min after logging in, set old maintenance players to 0, and grouped data into 5 minute intervals
- Converted
setInterval for updating player/review dashboard to use exact setTimeout preventing new data update timing drift on Firefox
- Added logic to refresh review chart 4 minutes after the hour
9/18/2025
- Created callout cards for steam reviews based on user zoom window (allows user to drill down to windows of their choice and inspect statistics)
- On average, in recent years most users do play more than 10 days before leaving a review
- Added About page (this one!)
9/17/2025
- Changed review scraping logic to use
requests.Session and changed to hourly job
- Fixed bug in review scraping logic where
cursor wasn't reset to * after finishing
- Fixed bug in current player count when
null
9/16/2025
- Refactored JavaScript into ES6 modules, previously 500 lines in 1 file
- Fixed bug where dashboard updating logic added new data to the array and HighCharts; HighCharts mutates original array—only need to
addPoint on HighCharts
- Discovered bug in scraping Steam reviews after the first scrape
9/15/2025
- Automated setting player counts to
null when server is offline
9/14/2025
- Added weekly player count change card
- Created daily job to scrape Steam reviews
- Deployed dual-axis steam review chart: % of total positive reviews and number of reviews
- Designed logic to make dashboard update every minute
- Created tooltips on hover to explain potential questions—keeping dashboard design clean
- Made frontend mobile friendly with media queries and alternate navbar
- Fixed bug where the player count chart wouldn't shrink on mobile due to callout card elements not shrinking
- Fixed bug in weekly difference calculation
- Released dashboard to public!
9/12/2025
- Created callout cards for current player count, server maintenance status
- Designed fixed floating navbar for easy navigation
- Designed skeleton loading for callout cards due to >750ms load times (webserver is in EU)
9/10/2025
- Began working on frontend design
- Deployed first chart displaying player counts every minute using HighCharts
- Created automated job to check if the game is undergoing maintenance, ground truth for 0 players online
- Decoupled data collection and database logic
9/9/2025
- Started looking at the data; reliable source but faulty during maintenance times
- Wrote basic script to automate filling in incorrect player counts and maintenance times with
null and adding missing minutes when my collector was offline
9/6/2025
- Set up basic webserver with Flask and nginx, made basic landing page
- Connected webserver to SQLite database
- Learned
mv deletes files when moving into directories without a trailing slash
8/30/2025
- Created automated job to gather player count in SQLite; gathers data every minute
- Added basic logging functionality
