As Ben mentioned in our previous post, our top priority right now is improving the stability of The Old Reader. To start, we’re going to get The Old Reader a much needed hardware upgrade. This week, we’ll be relocating the application to a top tier host located in the United States, tripling database capacity and adding over 10 times the network capacity.
The move is going to entail exporting all of the posts from about 6 million subscriptions, moving that data approximately 5000 miles and then importing it into the new database servers. This is a big move, and unfortunately it’s going to require about 48 hours of downtime.
The new environment will be ready to roll on Tuesday at which point we will begin the transfer and maintenance window. We’re shooting to begin that maintenance window at approximately 12AM GMT Wednesday. During this time, we’ll be frequently updating Twitter, Facebook and Status page to make sure you know as soon as it is back up and running.
We really do apologize that we’ll be down for so long. We’re avid users ourselves, and a couple days without The Old Reader is pretty tough for everyone. However, on the other side of this migration lies the stability and capacity that our favorite reader truly needs. Thank you for using The Old Reader and for your incredible patience.
We’re pleased to announce that The Old Reader will officially remain open to the public! The application now has a bigger team, significantly more resources, and a new corporate entity in the United States. We’re incredibly excited to be a part of this great web application and would like to share some details about its future as well as thank you for remaining loyal users. We’re big fans and users of The Old Reader and look forward to helping it grow and improve for years to come.
First off we want to say that it’s rare to have an application that inspires as much passion as The Old Reader has as of late. We think that’s a sign of greatness and all credit for that goes to the wonderful team that has been running the show including Dmitry and Elena. We’ve gotten to know them pretty well this past week and they are smart, honest, and passionate people. We’re happy to announce that they are still a part of the team and we hope they will be for a long time to come. The new team will be managing the project and adding to the engineering, communications, and system administration functions.
So now for the future. The Old Reader is going to retain all of its functionality and remain open to the public. Not only that, we’re going to do everything in our power to grow the user base which will only accentuate the things that make this application special. To facilitate these improvements, we’re going to be transitioning The Old Reader to a top tier hosting facility in the United States this coming week. It’s going to require some downtime and for that we sincerely apologize, but it’s also going to mean A LOT more servers, 10x faster networks, and long-term stability. We realize that doesn’t make the downtime easy but rest assured that things are looking up.
Over the coming weeks we’ll talk more about the new team of The Old Reader. We’re looking forward to introducing ourselves and making significant improvements to this incredible application. Thanks for reading and thanks for using The Old Reader!
UPD: We have received a number of proposals that we are discussing right now. Chances are high that public The Old Reader will live after all
Since we launched first public version almost a year ago up until March 2013 we have been working on The Old Reader in “normal” mode. In March things became “nightmare”, but we kept working hard and got things done. First, we were out of evenings, then out of weekends and holidays, and then The Old Reader was the only thing left besides our jobs. Last week difficulty level was changed to “hell” in every possible aspect we could imagine, we have been sleep deprived for 10 days and this impacts us way too much. We have to look back.
The truth is, during last 5 months we have had no work life balance at all. The “life” variable was out of equation: you can limit hours, make up rules on time management, but this isn’t going to work if you’re running a project for hundreds of thousands of people. Let me tell you why: it tears us to bits if something is not working right, and we are doing everything we can to fix that. We can’t ignore an error message, a broken RAID array, or unanswered email. I personally spent my own first wedding anniversary fixing the migration last Sunday. Talk about “laid back” attitude now. And I won’t even start describing enormous sentimental attachment to The Old Reader that we have.
We would really like to switch the difficulty level back to “normal”. Not to be dreaded of a vacation. Do something else besides The Old Reader. Stop neglecting ourselves. Think of other projects. Get less distant from families and loved ones. The last part it’s the worst: when you are with your family, you can’t fall out of dialogues, nodding, smiling and responding something irrelevant while thinking of refactoring the backend, checking Graphite dashboard, glancing onto a Skype chat and replying on Twitter. You really need to be there, you need to be completely involved. We want to have this experience again.
That’s why The Old Reader has to change. We have closed user registration, and we plan to shut the public site down in two weeks. We started working on this project for ourselves and our friends, and we use The Old Reader on a daily basis, so we will launch a separate private site that will keep running. It will have faster refresh rate, more posts per feed, and properly working full-text search — we are sure that we can provide all this at a smaller scale without that much drama, just like we were doing before March.
The private site?
Accounts will be migrated to the private site automatically. We will whitelist everybody we know personally, along with all active accounts that were registered before March 13, 2013. And of course, we will migrate all our awesome supporters and people who donated to keep the project running (if you sent us bitcoins, please get in touch to get identified). Later this week your account will get a distinct indication whether it will be migrated to the private site or not. If you see that message and believe that it’s wrong, or if all your friends are getting migrated and you are left behind — please, drop us a line.
Give me my data!
You will have two weeks to export your OPML file regardless of our decision. OPML export link is located at the bottom of the Settings page — use the top-right menu to get there. All posts that you saved for later by using Pocket integration will obviously remain in your Pocket account.
But you could…
For those who would like to start the usual “VC, funding, mentor” or “charge for the damn thing” mantras — please, spare it. We’re not in the Valley where it might be super-easy, and, after all, not everyone wants to be an entrepreneur. We just love making a good RSS reader.
We really want The Old Reader to be a big and successful project, with usable free accounts. But this is not possible to achieve with what we have, so unless someone resourceful takes over the project and brings it to the next level, it is not gonna happen. We had over 2 000 new registrations after the blackout last week. This is amazing and sad at the same time.
If anyone is interested in acquiring The Old Reader and making it better, we are very open and accepting proposals at email@example.com. We would be waiting for them for two weeks, supporting and maintaining The Old Reader as usual. Please don’t write us if you don’t have resources to maintain a site used by tens of thousands of people every day, or if you don’t know how you would improve The Old Reader. And please spare our time if you just want to buy the domain name and park a bunch of silly ads there — it’s not going to happen.
We value our community very much, and we will either pass the project to somebody who we know is going to take a good care of it, or we will switch it to private mode.
From one point of view, it’s not a big deal: “RSS is obsolete”, nobody died, we don’t owe anybody anything, you name it. Also, there are a lot of good readers around to choose from, a large part of them is smaller than The Old Reader and had not experienced growing pains of 80 000 daily active users in no time. But for us, it’s heartbreaking.
I will finally get back to work on my small studio — Bespoke Pixel — which has been run by my awesome partner all this time. Dmitry will keep being bright young software developer, making scalable and beautiful projects. Our team will stay together, and will keep working on making the private version of The Old Reader awesome.
We feel great responsibility for the project. We’d rather provide a smooth and awesome experience for 10 000 users than a crappy one for 420 000.
Sorry, each and everyone if we failed you. You are an incredible, supportive and helpful community. The best we could possibly hope for.
All the love, Elena Bulygina and Dmitry Krasnoukhov
On Saturday (July 20) we moved over a terabyte of data from one storage system to another. We made the move because the amount of data we have to store simply did not fit on our servers, and our preliminary tests showed that the new system will only be using about ⅓ of the disk space.
Migration went through fine, however we started seeing a higher I/O load after we finished, and suddenly one SSD drive in one of our database servers stopped working. Not a big deal, we thought — we obviously store data in mirrored mode on several servers — so we asked our hosting provider to swap the drive. We had to re-sync the data to the new disk, so you might have noticed the site being flacky.
However while data was syncing, another SSD drive went down in a different server. We got it replaced, and started syncing two database shards at the same time. At this point we lost 2 more drives, one of which unfortunately was on the server in one of the shards that were repairing.
While you can fly a plane with an engine off, unfortunately when all your engines stop your flight is over. Right now we have to restore our last pre-migration full database backup, and apply incremental updates to bring database to a fresh and (hopefully) consistent state.
Here comes the worst news - this will probably take a day or two.
Sorry about that.
This is a tough and incredibly stressful situation, but it looks like we have no other choice. We understand your frustration (actually, we are in the same boat: we are RSS junkies and built The Old Reader for ourselves and friends) and we are doing everything to make it as fast and painless as possible and live happily ever after.
After that, we will deploy bug fixes along with new things and improvements we have already developed. During last year we adapted and successfully expanded first from 2000 to 5000 users practically overnight, then from 10 000 to 160 000 in several weeks and from 200 000 to 400 000 in four months, so we are considering this as a new level-up for the project (although bumpy and painful one).
It’s 5 AM right now and backup restoration has already begun. We are monitoring and working on The Old Reader nearly 24 hours a day. We will keep frequent (but not annoying updates) in Twitter and will answer all your questions.
We deeply apologize for what has happened but we intend to come back in a much better way.
Thank you very much for your patience, support and understanding,
The Old Reader team.
July 25, 21:07 UTC
Back online! We hope this outage lasting July 25 19:12 UTC
If everything goes as planned, we should be back in 4-5 hours.
July 25 15:50 UTC
Import — check;
Indexes — check;
Balancing data between shards and configuring replicasets — in progress.
July 25 08:45 UTC
It looks like we have managed to upload the data. If indexes get generated correctly, we might be back online later today.
July 25 2:00 UTC
Continuing the upload, hoping it goes as planned, counting hours.
July 24 14:00 UTC
Proceeding with restore. More details hopefully in the evening.
July 23 18:00 UTC
We have managed to create a consistent dump of our database and started uploading it to the database servers.
Also, check out the Toggle Sidebar bookmarklet by Chas J. Owens. We love how our apps page is growing and are looking forward to see the first Windows Phone application join the club.
And good news everyone: the Paypal button on our Donate page is back. Thank you all for your support and incredible patience. Again, a friendly reminder: if you are on a tight budget, don’t send us anything.
Maintenance alert: The Old Reader will be unavailable for some time this Saturday. We will be migrating to a different storage system that will allow us to have much, much more data on the same number of servers. This means that we will be able to increase the number of posts we store for each feed, and make the inactivity period longer, which has been voted the most wanted feature by you, our users.
We will start working at 18:00 UTC, and we expect this maintenance to take about 6-8 hours (data, data everywhere). We have chosen the time with the lowest traffic to affect as few of you as possible. Please enjoy this Saturday break; your RSS feeds will be waiting for you on Sunday morning.
Spoilers! If everything goes as planned, we will be deploying lots of fixes and some new features shortly. See you next week!
Last week we launched API that has been very well accepted. We’ve got lots of feedback, but more importantly there already are two wonderful apps supporting The Old Reader: gReader for Android and Feeddler for iOS. For those who are into open source and Linux, feel free to check out Liferea. And the good news is: there are several more to come (can’t wait to share them with you). We now have a separate page listing all apps, extensions and related stuff. This week we were busy adjusting our infrastructure, fixing some bugs, and tweaking API so that more mobile apps could join the gang.
The first post-Google-Reader week has almost finished. We had been both afraid and excited about it, but it turned out not that bad. Let’s check our favorite graph of registered users:
The first graph covers time period from March until May and should give you a picture of what The Old Reader was before, and how we had to grow to handle the first wave of soon to be Google Reader refugees. The second one is relatively recent, from May till July 5 where you can see things heating up, but still not nearly half as much as in March.
Today we had our first major outage that can be perfectly described by infamous Murphy’s law: “anything that can go wrong, will go wrong”. Because we use this amazingly cheap but somewhat unreliable hosting provider, we had some issues with our database servers. Unfortunately, it happened at 3 am (or at 5 am for another half of the team). Usually we are always oncall and keep an eye on The Old Reader for ~20 hours a day, but this fell precisely into our blind spot.
We have set up some additional notifications, so hopefully we will be alerted immediately if something like this happens again. Because sleeping is definitely overrated and reading RSS is not. Sorry about what happened today. We will do better.
Next, community management time! Some of your comments and suggestions can be answered in a collective way:
1. Charge for the damn thing! / What is your business model? We are currently community-funded. You can support us using Flattr or Bitcoins. Most likely, we will bring back the PayPal button next week.
As for the future business model: we decided to go with freemium, and we are sticking to that. It’s a task of two parts: first, there are legal and administrative issues. In terms of these, we are probably that anecdotal deaf, three legged, one-eyed, and half-paralyzed dog called Lucky, but it looks like everything is nearly done. Second part is actually coding premium features and integrating with payment processing. We are still to do that.
We aim the site to be completely usable for free users, and we want paid users to get an even better experience (later improved to super awesome experience). We are gradually getting there.
2. Next Item Bookmarklet Chas. J. Owens made one. Thank him. It requires some effort to setup, but the instructions are clear.
Good news everyone. It looks like we can finally show some result of what we have been doing for the last couple of months, namely our mobile API. We have been running it on our test server for a week, gathered some initial feedback from several awesome mobile developers, and we think it’s now ready to be tested by the general public.
What does it mean to you? It means that there is now technical possibility to use The Old Reader with your favorite RSS app.
You can already try The Old Reader in Feeddler – free version already supports it, and the Pro version should get updated any time soon. Kudos to Che-Bin Liu for being extremely helpful in testing the API and getting Feeddler integrated so fast.
The documentation for the API is available in a separate github repo. It’s not the best piece of docs we’ve ever seen, but it seems to cover the basic use cases. You are welcome to improve it, just send us a pull request with your adjustments. Please note that even though you can use API both via http and https, we highly encourage you to use https for security reasons.
And last, but not least, spread the word. Let the developer of your favorite RSS app know about The Old Reader API, and ask them to get integrated. We would really like to see more and more apps working with the site bringing native mobile experience to the users of all platforms.
Beautiful and unrelated picture from Trending page. By Michal Karcz
So, many users have been asking us about our plans and some ETAs for many features we have in our roadmap. As all of you probably know, The Old Reader is something that we do in our spare time, and obviously the amount of work we can do is heavily influenced by our day jobs, personal life, and air dates of our favorite TV shows. So while we usually are quite open about our plans, we prefer not to promise any release dates.
However, today is different. One of the most voted feature requests in our roadmap (and an important pre-requisite for the most popular request) – API – is nearly ready. We finished what we believe is the first beta of the API, and we have started getting some preliminary feedback from a small group of early testers. Our plan is to get it deployed to production next week. If you are a mobile app developer, and you’d like to participate in the early testing – please get in touch with us, and we will be happy to give you access to our testing playground.
What’s next? Well, given that Google Reader will cease to exist in less than 2 weeks, we have been seeing quite a lot of new users recently, so we need to spend some time on refactoring our data storage systems to be able to serve this increasing number of users with the limited resources we have. Why not just buy more servers, you’ll ask? Well, we’d love to, but the amount of donations we receive right now covers about half of our hosting cost, so we have to pay several hundreds of Euro every month to run the site. We almost finished all the legal formalities, so we hope to be able to get the Paypal donations button back pretty soon. It feels like making The Old Reader was easier than registering a company to accept payments to run it. This is ridiculous.
Yeah, and since we mentioned Google Reader, lots of new users is not the only indicator of day X getting closer and closer. It looks like during the past few weeks every other person with coding skills decided to make their own RSS reader. We love that — the more the merrier. But we are not amused with the some marketing techniques used. We have been getting reports that users who read our updates in twitter are getting mass followed by some of the new kids on the block. We’d like to apologize to all our users who became a victim of this spam. Sorry, guys, gals and cats, we don’t like it as much as you do.
Our first public beta was released on June 13 a year ago. It had an ugly icon and default Bootstrap theme, but it was ok. At least, our friends said so. Looking back, I can’t believe what we have actually achieved in a year, and I absolutely can’t believe the amount of things that need to be done. The scale of things is also kind of scary.
One of our users suggested that we update our blog more often, another one posted the picture below, so we decided to write more regular updates of what we are up to. And even though The Old Reader is our free time project, we usually spend time on it every day.
Google Reader is due to be closed in less than a month, and our team can feel things heating up: new web-based RSS readers seem to be launching every week; we got some new code contributors aboard; and we started getting more and more questions about API and mobile apps.
Well, API is being cooked, and it is getting there. It is about 70% done, and I can already have first The Old Reader in-app experience (basic and painful, but still!). We really hope we can completely finish it in July, and it remains our top priority. I already have a mailing list for developers (thank you for your input on REST vs Google Reader API) and everyone else who expressed their interest in this topic, so the moment it’s ready we’ll spread the news as much as we can.
Besides working in API, we’ve been quite busy with other stuff during last couple of months: improved our feed refresh intervals significantly, fixed a number of nasty bugs that duplicated user posts in some cases, improved monitoring, and answered what feels like several thousands of emails, forum messages, and tweets.
We have also recently enjoyed crossing another milestone in user count – 218 – and sorted some legal issues that seemed actually more difficult than all our technical challenges combined.
We also launched several new servers so that we could fit new users, and our hosting cost increased even more. So we will kindly remind you of that Flattr button (hopefully it won’t be there for too long).
Next planned post will be about our future plans, making The Old Reader sustainable, and probably some announcements.
(image from our Trending section which never fails to amuse)