Hello,
For those of you who have setup your own server, what should I be expecting to pay each month through something like AWS?
Thanks in advance,
Ben
1 Like
It highly depends on what you want to achieve with your own server.
I use my mirror server of MB only for development purposes and only internal.
There are only the power costs for a virtual maschine.
If you want to let hundreds or thousands of people access your public accessible mirror server, I assume a much higher sum (including all your security and control tools around the server itself).
Maybe someone else is using such an environment?
1 Like
It very much depends on your setup, and what your use-case is.
Depending on the route you take, you could build a complete custom solution on top of Postgres and AWS ECS. Otherwise, if you clone and work with the MB Server from Github you will need a large file system like AWS elastic file system for data dumps, data seeding, etc. They have a Docker example, but our team ended up rolling our own setup on AWS. It’s under 60 USD a month for us. The trade-off is we are on the hook for managing a lot of the data streams, intake, potential downtime, etc.
1 Like
May I ask what the main purpose of your own setup is?
1 Like
Thanks everyone.
For context, I want to use MusicBrainz as the primary metadata source for my app, icheckmusic.co. Currently, I import releases from Discogs into my Mongo database as needed. The logic for importing is becoming complex which is why I want to switch to a MusicBrainz mirror server.
Although I’m unsure of my current database request volume, my app maintains around 50 weekly active users. I’m leaning towards a Docker setup for its apparent ease, especially since I’m new to Docker and managing production servers.
I would greatly appreciate any insights on the following:
- Estimated costs. @Firedance you mentioned costs being $60 per month. I’m curious what kind of traffic you have and if you have an idea of what that cost would translate to if using the Docker setup.
- Advice on deploying the Docker container on AWS, specifically what service I should use.
3 Likes
Sure, we are a tech. firm that deals with scaling large healthcare data streams.
Our back-end team took on MusicBrainz as a “hack-a-thon” type project to learn how to properly scale Postgres independent of AWS Aurora, and ended up using the basis for their work on a large enterprise system we developed for heart-rate monitoring. The primary interest was data ingestion on a rolling schedule, with fast just-in-time processing of the context of the data. The team here used MusicBrainz to practice rolling out an origin and replica boxes on Docker, and it was highly successful much in part thanks to the MusicBrainz software they developed with. Other pieces of technology handled the processing of the context, such that the system they developed could (briefly) summarize a large portion of changes from a nightly mbdump. This stack was then applied to real-time medical data analysis.
5 Likes
Sure these are great questions, but again it varies on your use-case and the technical capabilities of your development team.
On our non-MusicBrainz project, our traffic is heavy (100k requests/12h period), but the ECS cost and setup is still minimal, under ~$60 a month. Where AWS makes their money is services like Lambda, and managed setups like Aurora or scaling out multiple massive DynamoDB tables. These services eventually end up relying on other AWS offerings, such as RedShift, DataIngestion/etc, to make ends meet for scale.
A good analogy is if one were to setup something similar to the early days of Pandora-type infrastructure, which also used MusicBrainz at the beginning. The setup requires a lot of the aforementioned technologies working together. A combination Postgres + Logstash + OpenSearch/ElasticSearch/SOLR + a server(s)/container(s) to manage all of these processes.
For deploying Docker, I always recommend AWS ECS. You can choose AWS Fargate if you don’t want to manage your own servers, but if you need more fine-tuned control over your servers you can also provision EC2 resources to run your Docker containers; although you have to scale them yourself. EKS (Kubernetes alternative) is available as well, if you end up going that route.
1 Like
Thank you for the context and advice! This is very helpful, and now I have a better idea on how to move forward.
-Ben
1 Like