We’re excited to announce that starred items are now live in The Old Reader. This has been one of the most requested features and something we’ve felt belongs in the application for a long time. Hotkey (f) and API support are also available. Starred items will automatically be sent to pocket for users that have it activated.
As most of you know, our focus over the past few months was to increase performance and stability of The Old Reader. We’ve made tremendous strides and can now focus on adding functionality and making this tool a long-term sustainable platform built for the Open Web. The best is yet to come.
This post is not about the day to day operations of The Old Reader or anything of that nature. It’s about how our team came to get involved with RSS and how we see the future of this application and technology that we value so highly.
As a long time user of RSS and Google Reader, I’ve long appreciated the benefits of the technology. Like many people, my use of Google Reader faded a bit as social media platforms took hold. But, I’d always go back to Google Reader when I wanted to cut through the noise of social networks and focus on things I’m really passionate about. Google Reader wasn’t my “second screen” application where I’d go to take a break from work. It filled a much more essential need for me by providing these three features:
1. Unread items are kept in a queue. I don’t miss things. No algorithm chooses what to show me or not show me.
2. It’s an archive of blogs that I value and posts that I’ve read.
3. I can follow whatever I want from anywhere on the web. It embodies the openweb.
For my professional career in web research and development, I can’t really live without these features. I can follow twitter feeds or like Facebook pages, but I’m certain to miss important content from people who I highly value. I need those items queued, archived, and I need to be able to subscribe to anybody on the entire openweb. I can’t be limited to those authors who choose to enter into private social networks and I don’t want to have to constantly check my accounts for updates.
So this leads me to how we got involved in The Old Reader. When Google Reader shut it’s doors, my business partner Jim did some research and tried several services and suggested I’d like The Old Reader the best. So we both moved on over. I read some articles trying to understand why Google Reader would shut down and one really stuck with me. It hypothesized that Google had been following the lead of companies like Facebook and Twitter by turning their backs on the openweb and trying to build their own private/closed social networks. It’s frankly hard to argue against this theory.
However, we see this trend of migrating from the openweb to private networks as cyclical. How long will it be before your Facebook stream is so full of promoted content, bizarre algorithmic decisions, and tracking cookie based shopping cart reminders that you won’t be getting any valuable information? For as little as $60, a business can promote a page to Facebook users. It won’t be long before your news feed is worthless. So we jumped at the opportunity to get involved with developing and managing The Old Reader. We believe in it.
As we’ve been looking to grow our engineering team at Levee Labs and The Old Reader we’ve met with a number of bright young people that are surprisingly unaware of RSS. They say “I recognize the RSS icon, but haven’t really ever used it.” Is it possible that there is a lost generation of internet users that are completely unfamiliar with RSS? Are they unfamiliar with the idea of the openweb too? We believe that’s the case and we’ve been working hard to come up with ideas that’ll expose that generation to RSS, The Old Reader, and the openweb. It’s what made the internet great to begin with and it’s coming back.
We’d like to take a minute to thank the community of applications developers that have assisted in improving the user experience and accessibility of The Old Reader. Please take a moment to browse the list and let us know if there is anything we’re missing. New apps are still being added and developed specifically for The Old Reader and we’re thrilled to say that other prominent applications have us on the shortlist to be supported soon.
Along these same lines, please know that all of your feedback and voting at UserVoice is appreciated and something we take very seriously. Our first priority has been to transition The Old Reader and improve the architecture and performance. As many of you have noted, we’ve made huge strides in that area. In the near future, we plan to communicate more around specific features and begin making functional improvements.
We just wanted to say a quick hello and thank everybody for continuing to use The Old Reader. We’ve been hard at work making improvements and planning for the future. Thanks to everybody who has reached out to us, we’re working to get back in touch with each of you but please be patient with us during this busy time. We have some changes in store as we work to push this application into a long term sustainable position. We’ll be in touch soon.
As Ben mentioned in our previous post, our top priority right now is improving the stability of The Old Reader. To start, we’re going to get The Old Reader a much needed hardware upgrade. This week, we’ll be relocating the application to a top tier host located in the United States, tripling database capacity and adding over 10 times the network capacity.
The move is going to entail exporting all of the posts from about 6 million subscriptions, moving that data approximately 5000 miles and then importing it into the new database servers. This is a big move, and unfortunately it’s going to require about 48 hours of downtime.
The new environment will be ready to roll on Tuesday at which point we will begin the transfer and maintenance window. We’re shooting to begin that maintenance window at approximately 12AM GMT Wednesday. During this time, we’ll be frequently updating Twitter, Facebook and Status page to make sure you know as soon as it is back up and running.
We really do apologize that we’ll be down for so long. We’re avid users ourselves, and a couple days without The Old Reader is pretty tough for everyone. However, on the other side of this migration lies the stability and capacity that our favorite reader truly needs. Thank you for using The Old Reader and for your incredible patience.
We’re pleased to announce that The Old Reader will officially remain open to the public! The application now has a bigger team, significantly more resources, and a new corporate entity in the United States. We’re incredibly excited to be a part of this great web application and would like to share some details about its future as well as thank you for remaining loyal users. We’re big fans and users of The Old Reader and look forward to helping it grow and improve for years to come.
First off we want to say that it’s rare to have an application that inspires as much passion as The Old Reader has as of late. We think that’s a sign of greatness and all credit for that goes to the wonderful team that has been running the show including Dmitry and Elena. We’ve gotten to know them pretty well this past week and they are smart, honest, and passionate people. We’re happy to announce that they are still a part of the team and we hope they will be for a long time to come. The new team will be managing the project and adding to the engineering, communications, and system administration functions.
So now for the future. The Old Reader is going to retain all of its functionality and remain open to the public. Not only that, we’re going to do everything in our power to grow the user base which will only accentuate the things that make this application special. To facilitate these improvements, we’re going to be transitioning The Old Reader to a top tier hosting facility in the United States this coming week. It’s going to require some downtime and for that we sincerely apologize, but it’s also going to mean A LOT more servers, 10x faster networks, and long-term stability. We realize that doesn’t make the downtime easy but rest assured that things are looking up.
Over the coming weeks we’ll talk more about the new team of The Old Reader. We’re looking forward to introducing ourselves and making significant improvements to this incredible application. Thanks for reading and thanks for using The Old Reader!
UPD: We have received a number of proposals that we are discussing right now. Chances are high that public The Old Reader will live after all
Since we launched first public version almost a year ago up until March 2013 we have been working on The Old Reader in “normal” mode. In March things became “nightmare”, but we kept working hard and got things done. First, we were out of evenings, then out of weekends and holidays, and then The Old Reader was the only thing left besides our jobs. Last week difficulty level was changed to “hell” in every possible aspect we could imagine, we have been sleep deprived for 10 days and this impacts us way too much. We have to look back.
The truth is, during last 5 months we have had no work life balance at all. The “life” variable was out of equation: you can limit hours, make up rules on time management, but this isn’t going to work if you’re running a project for hundreds of thousands of people. Let me tell you why: it tears us to bits if something is not working right, and we are doing everything we can to fix that. We can’t ignore an error message, a broken RAID array, or unanswered email. I personally spent my own first wedding anniversary fixing the migration last Sunday. Talk about “laid back” attitude now. And I won’t even start describing enormous sentimental attachment to The Old Reader that we have.
We would really like to switch the difficulty level back to “normal”. Not to be dreaded of a vacation. Do something else besides The Old Reader. Stop neglecting ourselves. Think of other projects. Get less distant from families and loved ones. The last part it’s the worst: when you are with your family, you can’t fall out of dialogues, nodding, smiling and responding something irrelevant while thinking of refactoring the backend, checking Graphite dashboard, glancing onto a Skype chat and replying on Twitter. You really need to be there, you need to be completely involved. We want to have this experience again.
That’s why The Old Reader has to change. We have closed user registration, and we plan to shut the public site down in two weeks. We started working on this project for ourselves and our friends, and we use The Old Reader on a daily basis, so we will launch a separate private site that will keep running. It will have faster refresh rate, more posts per feed, and properly working full-text search — we are sure that we can provide all this at a smaller scale without that much drama, just like we were doing before March.
The private site?
Accounts will be migrated to the private site automatically. We will whitelist everybody we know personally, along with all active accounts that were registered before March 13, 2013. And of course, we will migrate all our awesome supporters and people who donated to keep the project running (if you sent us bitcoins, please get in touch to get identified). Later this week your account will get a distinct indication whether it will be migrated to the private site or not. If you see that message and believe that it’s wrong, or if all your friends are getting migrated and you are left behind — please, drop us a line.
Give me my data!
You will have two weeks to export your OPML file regardless of our decision. OPML export link is located at the bottom of the Settings page — use the top-right menu to get there. All posts that you saved for later by using Pocket integration will obviously remain in your Pocket account.
But you could…
For those who would like to start the usual “VC, funding, mentor” or “charge for the damn thing” mantras — please, spare it. We’re not in the Valley where it might be super-easy, and, after all, not everyone wants to be an entrepreneur. We just love making a good RSS reader.
We really want The Old Reader to be a big and successful project, with usable free accounts. But this is not possible to achieve with what we have, so unless someone resourceful takes over the project and brings it to the next level, it is not gonna happen. We had over 2 000 new registrations after the blackout last week. This is amazing and sad at the same time.
If anyone is interested in acquiring The Old Reader and making it better, we are very open and accepting proposals at firstname.lastname@example.org. We would be waiting for them for two weeks, supporting and maintaining The Old Reader as usual. Please don’t write us if you don’t have resources to maintain a site used by tens of thousands of people every day, or if you don’t know how you would improve The Old Reader. And please spare our time if you just want to buy the domain name and park a bunch of silly ads there — it’s not going to happen.
We value our community very much, and we will either pass the project to somebody who we know is going to take a good care of it, or we will switch it to private mode.
From one point of view, it’s not a big deal: “RSS is obsolete”, nobody died, we don’t owe anybody anything, you name it. Also, there are a lot of good readers around to choose from, a large part of them is smaller than The Old Reader and had not experienced growing pains of 80 000 daily active users in no time. But for us, it’s heartbreaking.
I will finally get back to work on my small studio — Bespoke Pixel — which has been run by my awesome partner all this time. Dmitry will keep being bright young software developer, making scalable and beautiful projects. Our team will stay together, and will keep working on making the private version of The Old Reader awesome.
We feel great responsibility for the project. We’d rather provide a smooth and awesome experience for 10 000 users than a crappy one for 420 000.
Sorry, each and everyone if we failed you. You are an incredible, supportive and helpful community. The best we could possibly hope for.
All the love, Elena Bulygina and Dmitry Krasnoukhov
On Saturday (July 20) we moved over a terabyte of data from one storage system to another. We made the move because the amount of data we have to store simply did not fit on our servers, and our preliminary tests showed that the new system will only be using about ⅓ of the disk space.
Migration went through fine, however we started seeing a higher I/O load after we finished, and suddenly one SSD drive in one of our database servers stopped working. Not a big deal, we thought — we obviously store data in mirrored mode on several servers — so we asked our hosting provider to swap the drive. We had to re-sync the data to the new disk, so you might have noticed the site being flacky.
However while data was syncing, another SSD drive went down in a different server. We got it replaced, and started syncing two database shards at the same time. At this point we lost 2 more drives, one of which unfortunately was on the server in one of the shards that were repairing.
While you can fly a plane with an engine off, unfortunately when all your engines stop your flight is over. Right now we have to restore our last pre-migration full database backup, and apply incremental updates to bring database to a fresh and (hopefully) consistent state.
Here comes the worst news - this will probably take a day or two.
Sorry about that.
This is a tough and incredibly stressful situation, but it looks like we have no other choice. We understand your frustration (actually, we are in the same boat: we are RSS junkies and built The Old Reader for ourselves and friends) and we are doing everything to make it as fast and painless as possible and live happily ever after.
After that, we will deploy bug fixes along with new things and improvements we have already developed. During last year we adapted and successfully expanded first from 2000 to 5000 users practically overnight, then from 10 000 to 160 000 in several weeks and from 200 000 to 400 000 in four months, so we are considering this as a new level-up for the project (although bumpy and painful one).
It’s 5 AM right now and backup restoration has already begun. We are monitoring and working on The Old Reader nearly 24 hours a day. We will keep frequent (but not annoying updates) in Twitter and will answer all your questions.
We deeply apologize for what has happened but we intend to come back in a much better way.
Thank you very much for your patience, support and understanding,
The Old Reader team.
July 25, 21:07 UTC
Back online! We hope this outage lasting July 25 19:12 UTC
If everything goes as planned, we should be back in 4-5 hours.
July 25 15:50 UTC
Import — check;
Indexes — check;
Balancing data between shards and configuring replicasets — in progress.
July 25 08:45 UTC
It looks like we have managed to upload the data. If indexes get generated correctly, we might be back online later today.
July 25 2:00 UTC
Continuing the upload, hoping it goes as planned, counting hours.
July 24 14:00 UTC
Proceeding with restore. More details hopefully in the evening.
July 23 18:00 UTC
We have managed to create a consistent dump of our database and started uploading it to the database servers.