2022/11/14 12:39 UTC - Our systems team confirmed that this issue was due to the previously mentioned Kubernetes upgrade. Normally these are run outside of business hours, however in this instance the decision to run the upgrade earlier was made as there were no anticipated issues. We are now taking steps to ensure this doesn't occur again so as to avoid future problems.
13:05 UTC - Our systems team believes these issues occurred due to a Kubernetes upgrade in Google Cloud Platform's australia-southeast1 cluster starting at approximately 03:30 UTC. No downtime was expected during this upgrade, but it appears to have occurred due to a momentary traffic switch from old pods to new pods. We are still looking into why the upgrade took place during business hours in the AU region.
04:11 UTC - We believe this issue to be resolved currently. We are still investigating the cause(s) and will provide more information as soon as possible.
04:00 UTC - We are investigating issues affecting production stores in our Australia region. At this time, users may be unable to login and perform other key functions. We'll update this post with more information shortly. Thank you for your patience.
