State of things, Q2 2016

On some level I understand all the trouble MB is currently having (including server resources, search server, bugs, etc). Still from my perspective it seems like there’s no clear planning but almost like we are just waiting for next problem to arise. Most of the trouble shouldn’t have been a surprise. Now our editors are just annoyed with every update because while we get nice new features we usually keep breaking the old ones.

(EDIT: Instead of fully removing this let’s keep this here as a reminder. I should never post anything while being angry and/or frustrated and/or drunk :wink:

2 Likes

Most of the troubles we’re having haven’t really come as surprises. A lot of it is down to MetaBrainz having a very sparse economy. The MetaBrainz.org redesign last year has really helped to improve the cash flow, which has allowed us to hire @Zas as a system administrator. Until this time @Rob acted as system administrator (with some help from the developers), community person, business person, and god knows what other roles. With a bunch of this offloaded, we have a lot more (mental) capacity to actually plan ahead and be active instead of only being reactive. I’m sure @Rob will talk lots more about this, once he gets around to the 2015 Annual Report - but for now, I’ll invite you to follow the MetaBrainz IRC channel (either via the logs or by actually being there). There’s a lot of discussion going on at the moment (mostly between @Zas, @Rob, and @KodeStar (from FanArt.tv)) about our planned data center move. This is all way off topic for the current topic though.

TL;DR: We know. We’re working on it. Please bear with us a little while longer.

I’m not following this one though. What updates are we talking about here? What features have been broken?

(If you want to follow up to either of these, please do so in a new topic.)

2 Likes

I would like to applaud @bitmap, @Gentlecat, @chirlu and @zas for keeping the site up and running! The fact that we’re up and running despite the challenges we’ve had in the past few weeks, it is sheer amazing to me! Given all of the challenges we’ve faced in the past two years, I’m really happy that we haven’t had any serious failures. But, there is still plenty of time for those to his us before we’re out of the woods!

There are many things happening that can be a bit hard to see, but if you follow along in the MetaBrainz channel in IRC (or read the logs) you would have a better idea of where we stand and what challenges we face on a daily basis. Let cover a few topics to give some background on where we are:

Back in 2011 we were donated a pile of machines, even though I had already started making plans for hosting our stuff in the cloud. There were so many machines that I postponed the cloud plans and prepared the donated machines for service. That has carried us for 4+ years with almost no hardware cost! The plan was to move to the cloud sometime around 2015, but then I spent most of 2014/2015 dealing with team drama, putting us seriously behind schedule while our hardware decays.

Recently we’ve had some bad luck, however. We’ve also had some disrespectful commercial customers hit us really hard and we had to find and block them. We’ve had unexpected traffic spikes and when trying to address these unexpected traffic spikes, we had two more machines fail on us. Effectively dead, not working for more than an hour at a time. So, now we have the tough question: Do we buy expensive hardware that we might use for 6 months (~$5000) or do we try and save the money and tough it out?

Moving to a new hosting facility has an incredible number of dependencies that Christina (our Biz Dev manager), @zas and I have been working through. It may not seem that we have a plan, but we’re executing on the plan and we’re incredibly busy trying to make things happen to stop and talk about our plans. :frowning: To give you a taste of what we’re up against:

  1. We want to move our hosting to Europe and have a business presence in Europe in order to reduce the costs and inefficiencies of being solely as US based business. A lot of our traffic, customers and contractors are in the EU and it simply makes sense to have a presence here.

  2. To establish a presence in the EU I needed local help to help with the business matters as well as researching and establishing an EU organization. So I needed find a Biz Dev manager and that person is Christina.

  3. Once Christina was on board she researched our options for what was suited for us. To get that process moving, involved getting certified documents from California, board approval for spending funds to establish the organization, EU labor law research, (and we needed to swap a board member, too!), hiring help to establish the org and generally navigating the Spanish bureaucracy. (see the not really exaggerated https://www.youtube.com/watch?v=XXWZ3uAEKsw for some clues)

  4. Once the org had been established we need to convince the bank to open a bank account for us. The draconian US banking laws extend worldwide and the local bank had to ensure that they were not opening themselves up to thousands of $$$ in accounting hassles just to allow a tiny non profit to open a bank account. We finally have a bank account and will start paying people with it tomorrow!!

  5. At the same time we’re also working to set up an office for the growing team here in Barcelona. That required a byzantine process that barely started when you sign the lease. Getting power, internet and water setup has taken a frustratingly long time. Had I known how long, I would’ve stayed at my co-woring space for a while longer while addressing hosting issues.

  6. While Christina has been focused on the hard core paperwork, @zas is keeping the site running, which itself requires many heroics. @zas and I have started planning the move to the EU hosting provider. We’ve got a 5 page document that collects some of the open questions and requirements around this process: NewHost investigation - Google Docs . In a couple of weeks @bitmap and @zas will be here in Barcelona and we’re going to work on establishing a formal plan for moving to the new hosting company. The amount of work required to make this happen is making my head hurt.

  7. While Christina, @zas and I have our hands full, @bitmap and @gentlecat continue to release new features. If there are many regressions in this process, please respond to @freso’s request in a separate thread. We’d need to address that. Not to mention all the contributions from @freso and @reosarevok to keep the community happy and polite while we deal with less than optimal site conditions.

This is just a snapshot of everything that is happening behind the scenes that culminates with the goal of moving to a new hosting company and being setup in the EU. And mind you we’re doing this with a minuscule budget trying to save money left and right.

I think it isn’t fair to say that we’re have no plan and that we’re just sitting waiting for the next thing to break. The entire team is working extremely hard to keep things on an even keel. Given our current set of challenges I am exceptionally proud of my team.

I hope this gives you some insight to where we currently stand!

19 Likes

I trust your word on this and I’m just happy to be proved wrong. I agree I could have been more careful with selecting my words. It wasn’t meant to offend anyone. I hope it’s not too late to take my words back.

I’m a monthly donator, active MusicBrainz editor and a great fan of all MetaBrainz projects. I’m only frustrated about the current situation. And not that frustrated anymore after your reply to this topic. I appreciate the work done by the whole team and have nothing negative to say about any of them. I especially appreciate your time for explaining the current situation. Thank you!

10 Likes

Thanks – I appreciate your kind words. :slight_smile:

2 Likes