UPD: We have received a number of proposals that we are discussing right now. Chances are high that public The Old Reader will live after all
Since we launched first public version almost a year ago up until March 2013 we have been working on The Old Reader in “normal” mode. In March things became “nightmare”, but we kept working hard and got things done. First, we were out of evenings, then out of weekends and holidays, and then The Old Reader was the only thing left besides our jobs. Last week difficulty level was changed to “hell” in every possible aspect we could imagine, we have been sleep deprived for 10 days and this impacts us way too much. We have to look back.
The truth is, during last 5 months we have had no work life balance at all. The “life” variable was out of equation: you can limit hours, make up rules on time management, but this isn’t going to work if you’re running a project for hundreds of thousands of people. Let me tell you why: it tears us to bits if something is not working right, and we are doing everything we can to fix that. We can’t ignore an error message, a broken RAID array, or unanswered email. I personally spent my own first wedding anniversary fixing the migration last Sunday. Talk about “laid back” attitude now. And I won’t even start describing enormous sentimental attachment to The Old Reader that we have.
We would really like to switch the difficulty level back to “normal”. Not to be dreaded of a vacation. Do something else besides The Old Reader. Stop neglecting ourselves. Think of other projects. Get less distant from families and loved ones. The last part it’s the worst: when you are with your family, you can’t fall out of dialogues, nodding, smiling and responding something irrelevant while thinking of refactoring the backend, checking Graphite dashboard, glancing onto a Skype chat and replying on Twitter. You really need to be there, you need to be completely involved. We want to have this experience again.
That’s why The Old Reader has to change. We have closed user registration, and we plan to shut the public site down in two weeks. We started working on this project for ourselves and our friends, and we use The Old Reader on a daily basis, so we will launch a separate private site that will keep running. It will have faster refresh rate, more posts per feed, and properly working full-text search — we are sure that we can provide all this at a smaller scale without that much drama, just like we were doing before March.
The private site?
Accounts will be migrated to the private site automatically. We will whitelist everybody we know personally, along with all active accounts that were registered before March 13, 2013. And of course, we will migrate all our awesome supporters and people who donated to keep the project running (if you sent us bitcoins, please get in touch to get identified). Later this week your account will get a distinct indication whether it will be migrated to the private site or not. If you see that message and believe that it’s wrong, or if all your friends are getting migrated and you are left behind — please, drop us a line.
Give me my data!
You will have two weeks to export your OPML file regardless of our decision. OPML export link is located at the bottom of the Settings page — use the top-right menu to get there. All posts that you saved for later by using Pocket integration will obviously remain in your Pocket account.
But you could…
For those who would like to start the usual “VC, funding, mentor” or “charge for the damn thing” mantras — please, spare it. We’re not in the Valley where it might be super-easy, and, after all, not everyone wants to be an entrepreneur. We just love making a good RSS reader.
We really want The Old Reader to be a big and successful project, with usable free accounts. But this is not possible to achieve with what we have, so unless someone resourceful takes over the project and brings it to the next level, it is not gonna happen. We had over 2 000 new registrations after the blackout last week. This is amazing and sad at the same time.
If anyone is interested in acquiring The Old Reader and making it better, we are very open and accepting proposals at firstname.lastname@example.org. We would be waiting for them for two weeks, supporting and maintaining The Old Reader as usual. Please don’t write us if you don’t have resources to maintain a site used by tens of thousands of people every day, or if you don’t know how you would improve The Old Reader. And please spare our time if you just want to buy the domain name and park a bunch of silly ads there — it’s not going to happen.
We value our community very much, and we will either pass the project to somebody who we know is going to take a good care of it, or we will switch it to private mode.
From one point of view, it’s not a big deal: “RSS is obsolete”, nobody died, we don’t owe anybody anything, you name it. Also, there are a lot of good readers around to choose from, a large part of them is smaller than The Old Reader and had not experienced growing pains of 80 000 daily active users in no time. But for us, it’s heartbreaking.
I will finally get back to work on my small studio — Bespoke Pixel — which has been run by my awesome partner all this time. Dmitry will keep being bright young software developer, making scalable and beautiful projects. Our team will stay together, and will keep working on making the private version of The Old Reader awesome.
We feel great responsibility for the project. We’d rather provide a smooth and awesome experience for 10 000 users than a crappy one for 420 000.
Sorry, each and everyone if we failed you. You are an incredible, supportive and helpful community. The best we could possibly hope for.
All the love, Elena Bulygina and Dmitry Krasnoukhov
On Saturday (July 20) we moved over a terabyte of data from one storage system to another. We made the move because the amount of data we have to store simply did not fit on our servers, and our preliminary tests showed that the new system will only be using about ⅓ of the disk space.
Migration went through fine, however we started seeing a higher I/O load after we finished, and suddenly one SSD drive in one of our database servers stopped working. Not a big deal, we thought — we obviously store data in mirrored mode on several servers — so we asked our hosting provider to swap the drive. We had to re-sync the data to the new disk, so you might have noticed the site being flacky.
However while data was syncing, another SSD drive went down in a different server. We got it replaced, and started syncing two database shards at the same time. At this point we lost 2 more drives, one of which unfortunately was on the server in one of the shards that were repairing.
While you can fly a plane with an engine off, unfortunately when all your engines stop your flight is over. Right now we have to restore our last pre-migration full database backup, and apply incremental updates to bring database to a fresh and (hopefully) consistent state.
Here comes the worst news - this will probably take a day or two.
Sorry about that.
This is a tough and incredibly stressful situation, but it looks like we have no other choice. We understand your frustration (actually, we are in the same boat: we are RSS junkies and built The Old Reader for ourselves and friends) and we are doing everything to make it as fast and painless as possible and live happily ever after.
After that, we will deploy bug fixes along with new things and improvements we have already developed. During last year we adapted and successfully expanded first from 2000 to 5000 users practically overnight, then from 10 000 to 160 000 in several weeks and from 200 000 to 400 000 in four months, so we are considering this as a new level-up for the project (although bumpy and painful one).
It’s 5 AM right now and backup restoration has already begun. We are monitoring and working on The Old Reader nearly 24 hours a day. We will keep frequent (but not annoying updates) in Twitter and will answer all your questions.
We deeply apologize for what has happened but we intend to come back in a much better way.
Thank you very much for your patience, support and understanding,
The Old Reader team.
July 25, 21:07 UTC
Back online! We hope this outage lasting July 25 19:12 UTC
If everything goes as planned, we should be back in 4-5 hours.
July 25 15:50 UTC
Import — check;
Indexes — check;
Balancing data between shards and configuring replicasets — in progress.
July 25 08:45 UTC
It looks like we have managed to upload the data. If indexes get generated correctly, we might be back online later today.
July 25 2:00 UTC
Continuing the upload, hoping it goes as planned, counting hours.
July 24 14:00 UTC
Proceeding with restore. More details hopefully in the evening.
July 23 18:00 UTC
We have managed to create a consistent dump of our database and started uploading it to the database servers.
Also, check out the Toggle Sidebar bookmarklet by Chas J. Owens. We love how our apps page is growing and are looking forward to see the first Windows Phone application join the club.
And good news everyone: the Paypal button on our Donate page is back. Thank you all for your support and incredible patience. Again, a friendly reminder: if you are on a tight budget, don’t send us anything.
Maintenance alert: The Old Reader will be unavailable for some time this Saturday. We will be migrating to a different storage system that will allow us to have much, much more data on the same number of servers. This means that we will be able to increase the number of posts we store for each feed, and make the inactivity period longer, which has been voted the most wanted feature by you, our users.
We will start working at 18:00 UTC, and we expect this maintenance to take about 6-8 hours (data, data everywhere). We have chosen the time with the lowest traffic to affect as few of you as possible. Please enjoy this Saturday break; your RSS feeds will be waiting for you on Sunday morning.
Spoilers! If everything goes as planned, we will be deploying lots of fixes and some new features shortly. See you next week!
Last week we launched API that has been very well accepted. We’ve got lots of feedback, but more importantly there already are two wonderful apps supporting The Old Reader: gReader for Android and Feeddler for iOS. For those who are into open source and Linux, feel free to check out Liferea. And the good news is: there are several more to come (can’t wait to share them with you). We now have a separate page listing all apps, extensions and related stuff. This week we were busy adjusting our infrastructure, fixing some bugs, and tweaking API so that more mobile apps could join the gang.
The first post-Google-Reader week has almost finished. We had been both afraid and excited about it, but it turned out not that bad. Let’s check our favorite graph of registered users:
The first graph covers time period from March until May and should give you a picture of what The Old Reader was before, and how we had to grow to handle the first wave of soon to be Google Reader refugees. The second one is relatively recent, from May till July 5 where you can see things heating up, but still not nearly half as much as in March.
Today we had our first major outage that can be perfectly described by infamous Murphy’s law: “anything that can go wrong, will go wrong”. Because we use this amazingly cheap but somewhat unreliable hosting provider, we had some issues with our database servers. Unfortunately, it happened at 3 am (or at 5 am for another half of the team). Usually we are always oncall and keep an eye on The Old Reader for ~20 hours a day, but this fell precisely into our blind spot.
We have set up some additional notifications, so hopefully we will be alerted immediately if something like this happens again. Because sleeping is definitely overrated and reading RSS is not. Sorry about what happened today. We will do better.
Next, community management time! Some of your comments and suggestions can be answered in a collective way:
1. Charge for the damn thing! / What is your business model? We are currently community-funded. You can support us using Flattr or Bitcoins. Most likely, we will bring back the PayPal button next week.
As for the future business model: we decided to go with freemium, and we are sticking to that. It’s a task of two parts: first, there are legal and administrative issues. In terms of these, we are probably that anecdotal deaf, three legged, one-eyed, and half-paralyzed dog called Lucky, but it looks like everything is nearly done. Second part is actually coding premium features and integrating with payment processing. We are still to do that.
We aim the site to be completely usable for free users, and we want paid users to get an even better experience (later improved to super awesome experience). We are gradually getting there.
2. Next Item Bookmarklet Chas. J. Owens made one. Thank him. It requires some effort to setup, but the instructions are clear.