Recent changes at Crickles

If you’re a regular user of Crickles you’ll have noticed the new Pending page that arrived recently. This arose from a deep change to the way Crickles works. Previously, Crickles would take a once per day snapshot of all users’ activities and run a battery of calculations on them. Many of these calculations require analysis of activities over a long time period and comparisons between cohorts of users. Because of this, it’s not easy to update Crickles in real time. Furthermore, when new users sign up Crickles needs a relatively high amount of historical information as background for current metrics – for example, the Cardiac Stress Score for an activity requires a comparison with historical efforts.

Wherever possible, Crickles uses second-by-second (or near equivalent) data for heart rate and power from users’ sports devices as the basis for analysis. Where this is not available – for example, when the user doesn’t wear a heart rate strap or sports watch – we use machine learning algorithms instead. Getting all of this data for existing and especially new users from Strava requires Crickles to make a lot of calls to the Strava API; not unreasonably, Strava limits this. To an extent, Crickles has always tried to regulate API calls to avoid running into these limits but until recently at times of high load the limits would sometimes nonetheless by met and when this happened Crickles would use its machine learning algorithms in place of second by second data to derive CSS and related output. As a user, you’d see that in these cases CSS numbers would be there but not Normalised Power, Effort Spots and (if you have access to them) or Charts. Fundamentally, this did not affect core Crickles metrics but some users (including me) miss those extra features when they’re not there.

To avoid this Crickles now collects activity data in real time rather than on a daily cycle. This is how we’re able to produce the Pending page, although the extra data collection required for this new page is currently run periodically (generally within 30 minutes of upload) rather than in real time. In addition, there is now also forensic management of Crickles API calls to Strava so that we should never hit an API limit. The main calculation load that updates all of the pages in Crickles other than the Pending page is still done on the daily cycle but this now separated from data collection.

In short, a substantial amount of work has been done to implement changes that few users will notice! However, it’s a better, more robust architecture and it now remains open to Crickles to pull some of the calculations (such as Effort Spots) into near real time in future updates.

The changes were significant enough to risk disruption. I think everything is running smoothly now but if you see anything that looks awry please let me know.

Ian