We recently spotted an unusual bug that we believe was caused by a solar flare. In this post we'll document how the issue was identified and how it was resolved.
Image Credit: NASA/SDO
The morning of 14th October we received an alert from our telemetrics and then a helpdesk ticket following it advising us that one of React.JS UIs was not displaying correctly, the console presented an ‘Uncaught SyntaxError: Invalid or unexpected token’ error. At VI we use the a CI/CD system with Webpack for JS for our build and deploys, and this deploy was no different.
How we fixed it
We spun up a local version of the website; everything worked correctly. We next checked the JS files for changes and validated our supply chains to verify the differences between the local and production builds. This is all we found:
We couldn’t initially figure out what could possibly be causing this issue at this point, we thought it might be related to some of the Webpack modules we were using but as we verified nothing had changed, this idea didn’t stand up to scrutiny.
At this point we did not fully understand what was happening, so we did not ‘just try to deploy again’ to ‘see if it would fix it’, we instead placed the system into a ‘safe mode’ and worked the problem.
We suspected corruption in the build process and decided to place a check in our continuous integration system to validate compiled JS at the deploy stage (in addition to the test phase which already existed) using node --check. We verified this would have detected the previous issue, so we had confidence to now ‘release again’ with this check in place.
The release went through as normal, and soon the UI was behaving correctly and we closed the ticket and scheduled a retrospective.
Retrospective
We generally like to know why things go wrong to improve on our processes.
We identified a single character difference in the compiled JS to the local (and redeployed) build. Where we expected to see a Y we instead saw ⁙, which had no place in code.
After some digging, we identified 1 bit difference in the encoding of the Unicode characters Y and ⁙.
Also, as it happens one of our team spotted on Twitter a report of a minor solar flare on the day of the problem. When analysing solar activity we could also see that on this day there was an active solar storm:
We concluded that it was a solar flare that caused a single bit to flip while Webpack was running! Unusual for us indeed.
Even though this is an extremely rare event though we’ve still added a build step to our CI system that runs all compiled JS code with node --check flag. Hopefully that should stop any potential issues in the future caused by invalid syntax in JS files. At VI, even when issues aren’t caused by developers, we still make sure we improve our processes to increase the quality of the product we deliver to our clients.