First of all, we’re very sorry about the current issues and we’re working to fix them. All premium users will get time credited to their accounts.
Our primary syncing systems are hosted on Amazon Web Services (AWS). Earlier this week, AWS experienced significant downtime that caused Simplenote syncing to fail. The AWS downtime also affected sites like Reddit, Airbnb, and Foursquare.
We have a separate system for syncing notes with third-party apps (like Notational Velocity) that is currently hosted on Google App Engine (GAE). About three weeks ago, we started experiencing significant latency when writing changes to this system, resulting in delayed syncing ranging from a few minutes to a few hours. Our efforts to fix this on our own weren’t successful, so we turned to the GAE team for advice.
They recommended we immediately migrate our third-party syncing to a newer version of their platform. On Wednesday night, we decided to do this. It went less smoothly than we hoped. We did not properly take into account some consequences of the migration, but we corrected this last night.
This morning, GAE is experiencing a global outage and we are waiting to hear more news on that front. Our official iOS apps and all third-party apps that have upgraded to our latest API (like ResophNotes and DashNote) should be working since they don’t rely on GAE. Our web app is still hosted on GAE, so it will be down until their service is restored.
We’re a small team working hard to get things stable again. We’ll work on a process to remove duplicate notes that have appeared in some users’ accounts. When the dust has settled, we’ll write a more detailed post-mortem about our plans for improvement and how we’ll better handle situations like this in the future.
Simplenote 