GSOC 2024 : Integrate music streaming from Subsonic

About me

Name: Rana Satyaraj
Matrix Handle: Satyaraj
IRC Nickname: satyaraj
Location: Roorkee, Uttarakhand
Time Zone: UTC +05:30
e-mail: ranasatyaraj7103iitr@gmail.com
Github: SatyarajRana (Rana Satyaraj) · GitHub
Linkedin: https://www.linkedin.com/in/satyaraj-rana-751471230/

I’m Satyaraj Rana, currently a Junior at the Indian Institute of Technology Roorkee, Uttarakhand. I have been teaching myself coding for the past 2.5 years and have gained experience in Full Stack Web Development. When I’m not coding, I’m either playing Minecraft or playing some sport like Cricket, Table Tennis or Football. Also I’m deeply passionate about Cinema, my favorite one being Interstellar.

Why me?

My love for problem-solving. I think I’m a very resourceful guy. If you give me a problem, more than my skills, I trust my resourcefulness that I will come up with a solution. Additionally in my past two internships, I have gained substantial experience both in Development and in Communication and Teamwork.

In the past 2 months, I’ve familiarized with the Listenbrainz codebase, and although I’ve faced issues with setting up the environment on my laptop due to some zsh issues, with guidance from Monkey and Lucifer, I’ve been able to make some contributions to the project.

Here are my PRs:
https://github.com/metabrainz/listenbrainz-server/pull/2805
https://github.com/metabrainz/listenbrainz-server/pull/2765
https://github.com/metabrainz/listenbrainz-server/pull/2818

I am currently working on:
https://tickets.metabrainz.org/browse/LB-841

Proposed Project

Listenbrainz currently supports Spotify, Soundcloud and YouTube as the music discovery and streaming services. It would be great if the users were able to search and stream music on LB from their private music collection using apps like Subsonic, Navidrome, Funkwhale and other music apps that support the Subsonic API.

Project size: 350 Hours

Goals:

  1. Storing the HOST address of the service as well as user’s credentials by creating a Subsonic player service in the backend.

  2. Adding the react based frontend which allows the user to setup/edit their Subsonic service by providing the Host URL and credentials.

  3. Adding a new data-source: SubsonicPlayer to the BrainzPlayer and adding all the functionalities such as searching and playing tracks.

Implementation:

First, we have to handle the request when a user first tries to connect Subsonic as a service. For Subsonic service setup, we can have a form that takes the Username, Password and HostURL. This form will be in a new card at Connect Services. The UI for it should be fairly straightforward.

On submission of this form, the
music-services/<servce_name>/disconnect/’ route in the backend will be called with the Username, Password and HostURL in the request data. There are two different ways to handle this further:

  1. Call the add_new_user function inside the music_services_disconnect function itself.
  2. Redirect the user to to a checkSubsonic page with the credentials which will check the connection by pinging the Subsonic server, then call the ‘music-services/<service_name>/callback’ route and further call the add_new_user function

Either way, the add_new_user function must be defined.

A new class called SubsonicService has to be created which inherits from the ExternalService class, and add it to the ExternalServiceType enum, and accordingly update the getservice_or_raise_404 in settings.py.

class SubsonicService(ExternalService):
    def __init__(self):
        super(SubsonicService, self).__init__(ExternalServiceType.SUBSONIC)

    def add_new_user(self, user_id: int, username: str, password: str, hostUrl: str) -> bool:
        # Hashing the password using Subsonic_Key from config
        subsonic_key = current_app.config["SUBSONIC_KEY"]
        password_with_key = password + subsonic_key
        hashed_password = hashlib.md5(password_with_key.encode()).hexdigest()
        
        external_service_oauth.save_info(db_conn, user_id, self.service_type, username, hashed_password, hostUrl)
        
        return True

Additionally the following functions will also be added:

  • get_user()
  • revoke_user()
  • update_user()

In order to handle the database operations to store the username, password_hash and hostUrl, the following functions will be added either to the external_service_oauth.py file or in a new subsonic.py file in the db folder.

1 save_info
2 update_info

The save_info function can be defined as:

def save_info(db_conn, user_id: int, service: ExternalServiceType, username: str, password: str,
              host_url: str, record_listens: bool, scopes: List[str], external_user_id: Optional[str] = None):
    """ Add a row to the external_service_oauth table for specified user with corresponding info.

    Args:
        db_conn: database connection
        user_id: the ListenBrainz row ID of the user
        service: the service for which the info will be saved
        username: the username for the service
        password: the password for the service
        host_url: the URL of the host
        record_listens: True if user wishes to import listens, False otherwise
        scopes: the oauth scopes
        external_user_id: the user's id in the external linked service
    """
    result = db_conn.execute(sqlalchemy.text("""
        INSERT INTO external_service_oauth AS eso
        (user_id, external_user_id, service, username, password, host_url, scopes)
        VALUES
        (:user_id, :external_user_id, :service, :username, :password, :host_url, :scopes)
        ON CONFLICT (user_id, service)
        DO UPDATE SET
            external_user_id = EXCLUDED.external_user_id,
            username = EXCLUDED.username,
            password = EXCLUDED.password,
            host_url = EXCLUDED.host_url,
            scopes = EXCLUDED.scopes,
            last_updated = NOW()
        RETURNING id
        """), {
        "user_id": user_id,
        "external_user_id": external_user_id,
        "service": service.value,
        "username": username,
        "password": password,
        "host_url": host_url,
        "scopes": scopes,
    })

    if record_listens:
        external_service_oauth_id = result.fetchone().id
        db_conn.execute(sqlalchemy.text("""
            INSERT INTO listens_importer
            (external_service_oauth_id, user_id, service)
            VALUES
            (:external_service_oauth_id, :user_id, :service)
            ON CONFLICT (user_id, service) DO UPDATE SET
                external_service_oauth_id = EXCLUDED.external_service_oauth_id,
                user_id = EXCLUDED.user_id,
                service = EXCLUDED.service,
                last_updated = NULL,
                latest_listened_at = NULL,
                error_message = NULL
            """), {
            "external_service_oauth_id": external_service_oauth_id,
            "user_id": user_id,
            "service": service.value
        })

    db_conn.commit()

This also requires changes to be made in the ExternalService model. The updated model will look like this:

class ExternalService(db.Model):
    _tablename_ = 'external_service_oauth'

    id = db.Column(db.Integer, primary_key=True)
    user_id = db.Column(db.Integer, db.ForeignKey('user.id', ondelete='CASCADE'), nullable=False)
    # Workaround: cannot use Enum here because it seems SQLAlchemy uses the variable names of the enum instead of
    # the values assigned to them. It is possible to write a wrapper to change this behaviour but for our purposes
    # just using a string works fine so not going into that.
    service = db.Column(db.String, nullable=False)
    access_token = db.Column(db.String)              #nullable changed to true
    refresh_token = db.Column(db.String)
    token_expires = db.Column(db.DateTime(timezone=True))
    last_updated = db.Column(db.DateTime(timezone=True), server_default=db.func.now())
    scopes = db.Column(db.ARRAY(db.String))
    user = db.relationship('User')
    username = db.Column(db.String, nullable=True).  #New column for Subsonic
    password = db.Column(db.String, nullable=True).  #New column for Subsonic
    hostUrl = db.Column(db.String, nullable=True).   #New column for Subsonic

About the API:

The subsonic API calls require a Username, the MD5 hash of password, the key used to create the hash, and the server address.

Some of the methods we will be calling are as follows:
Note: Here,
fixed_parameters = u=&t=<password_hash>&s=&v=1.13.0&c=myapp&f=json
base_url = http://your-server/rest

  • Search:
    base_url/search2.view?fixed_parameters&query=${encodeURLComponent()}

  • Ping:
    base_url/ping.view?fixed_parameters

  • Stream:
    base_url/search2/.view?fixed_parameters&id=

Frontend:

In the frontend, we need to create a new SubsonicPlayer class which implements DataSourceType.
The functions of this class will require the subsonic credentials username, hostUrl and password hash. These will be stored as private variables in the class and will come from the prop: subsonicUser. This prop will be passed down by the BrainzPlayer, which will get it from the GlobalAppContext.
The global props are set in the get_global_props function in utils.py. We will add a new function get_current_subsonic_user function here which will be defined in view_utils.py. This function will call the get_user function inside the SubsonicService class in the backend.

Following are the main functions defined inside of the SubsonicPlayer class.

  • playListen: This function will have a helper function searchAndPlayTrack which will call a new searchForSubsonicTrack function defined in utils.tsx. This function calls looks like this:
const searchForSubsonicTrack = async(
  subsonic_server_url: string,
  subsonic_username: string,
  subsonic_password: string,
  susbonic_key: string,
  trackName: string,
): Promise<string | null> => {
  const response = await fetch(
    `${subsonic_server_url}/rest/search2.view?u=${subsonic_username}&t=${subsonic_password}&s=${susbonic_key}&v=1.13.
0&c=listenbrainz&f=json&query=${encodeURIComponent(trackName)}`
  );
  const responseBody = await response.json();
  if (!response.ok) {
    throw responseBody.error;
  }
  return responseBody?["subsonic-response"].searchResult2.song.id ?? null;
}

This function returns an id (of the song). The stream method will be called with this id, which will return an audio (blob) object that will be stored in a react state with the type HTMLAudioElement.

searchAndPlayTrack = async (listen: Listen | JSPFTrack): Promise<void> => {
    const trackname = getTrackName(listen);
    const {
      handleError,
      handleWarning,
      handleSuccess,
      onTrackNotFound,
    } = this.props;
    if (!trackName && !artistName && !releaseName) {
      handleWarning(
        "We are missing a track title, artist or album name to search on Spotify",
        "Not enough info to search on Spotify"
      );
      onTrackNotFound();
      return;
    }

    try {
        const track_id = searchForSubsonicTrack(
             this.subsonicUrl,
             this.subsonicUsername,
             this.subsonicPassword,
             this.subsonicKey,
             trackName
        );

        if (track_id) {
        this.playSubsonicId(track_id);
        return;
      }
        onTrackNotFound();
    }catch (errorObject) {
      if (!has(errorObject, "status")) {
        handleError(
          errorObject.message ?? errorObject,
          "Error searching on Spotify"
        );
      }
      if (errorObject.status === 401) {
        // Handle token error and try again if fixed
        this.handleTokenError(
          errorObject.message,
          this.searchAndPlayTrack.bind(this, listen)
        );
        return;
      }
      if (errorObject.status === 403) {
        this.handleAccountError();
      }
    }
}
playSubsonicId = async(
    track_id: string,
    retryCount = 0
): Promise<void> => {
    const {handleErorr} = this.props;
    if (retryCount > 3) {
      handleError("Cannot connect to the Subsonic Server");
      return;
    }
    if(!pingSubsonicServer){
       this.playSubsonicId.bind(this, track_id, retrCount+1);   
    }
    try{
        const response = await fetch(
    `${this.subsonicUrl}/rest/stream.view?u=${this.subsonicUsername}&t=${this.subsonicPassword}&
s=${this.subsonicKey}&v=1.13.0&c=listenbrainz&f=json&id=${track_id}`
  );
   
    const blob = await response.blob();
    const url = URL.createObjectURL(blob);
    setAudio(new Audio(URL));
    audio?.play();

    }catch (error){
        handleError(error.message, "Error playing from Subsonic");
    }
}

The audio react state can be defined as follows:

const [audio, setAudio] = React.useState<HTMLAudioElement | null>(null);

The play, pause, and currentTime methods can be called on this audio state to control the playback.

● seekToPosition: Seeking to a specific point in the song can be easily implemented using the currentTime property of the HTMLAudioElement.

● onTrackUpdated: The progress bar of the BrainzPlayer can be updated by adding an event listener timeupdate event of the HTMLAudioElement.

audio.addEventListener("timeupdate", () =>
      handleTimeUpdate(audio.currentTime)
    );

● onTrackEnded: The ending of a track can be detected by adding the ended event-listener to the audio element.

audio.addEventListener("ended", handleTrackEnded);

Project Timeline:

Pre-Community Boding period (April) :
I plan to discuss additional features like Importing listens with Lucifer and Mayhem, which for now, Lucifer has told me to exclude from the proposal. That would enable us to implement features like playlist imports as well.

Community Bonding Period (May) :
Create and finalize the UI MockUps for the Subsonic Service setup page. Also finalize the changes to be made in the external service schema with Lucifer.

Week 1:
In the backend, make the required changes in the external_service_oauth table schema. Also, update the code at places this change in schema affects.

Week 2:
Create the SubsonicService class, and define all its functions including the DB-related functions. i.e. save_info, update_info, etc. Also update the routes in settings.py that will call these functions.

Week 3-4:
Finalize the UI for the Subsonic server setup as well as code it in the frontend.

Week 5:
Create the new SubsonicPlayer component in the frontend and integrate it with the BrainzPlayer.

Week 6-7:
Define all the functions related to music playback in the SubsonicPlayer.

Week 8:
Write tests for the Subsonic class in the backend as well as the DB functions.

Week 9:
Write tests for the SubsonicPlayer in the frontend. Also I will discuss about secondary goals and additional functionality that need to be added.

Week 10-11:
This time will be dedicated to work on all the additional features apart from music playback that might be discussed. Also tidying up the code.

Week 12:
Buffer period, submitting my code for review.

I will keep the community updated about all the work on Monday meetings, as well as discuss the possibility of additional features, like listens Importing and any playlist related functionalities.

Community Affinities

What type of music do you listen to?

I have a wide variety of playlists in my Spotify account. Everything from EDM, Pop and Rock to Classical, Studio Ghibli, Lofi. I listen to a lot of Marshmello, Khalid and Coldplay songs.

Here are some of my favorite songs of all time:

Silence - Marshmello & Khalid : bb1f17e0-bdea-4816-9a2d-4819ad98f894

See You Again - Charlie Puth : 7b66c815-30fc-434e-b530-25e12f0b21c4

Father and Son - Cat Stevens : cf2d532a-ef95-4319-8315-835d355de249

Bohemian Rhapsody - Queen : b1a9c0e9-d987-4042-ae91-78d6a3267d69

Here is my ListenBrainz profile

What aspects of MusicBrainz/ListenBrainz/BookBrainz/Picard interest you the most?

I think ListenBrainz some really cool features like similar users and stats. It makes music discovery much better and also helps track all my music. Also, I’m loving the new Music-Neighbourhood feature!

Have you ever used any of our projects in the past?

I’ve been using ListenBrainz for a few months now.

Programming Precedents

When did I first start programming?

I started programming in my 1st year of college. I taught myself C and C++ initially. Also learned the fundamentals of Data Structures and Algorithms. Then I got into Web Development, initially with the MERN stack. Later on, I made myself familiar with Python, SQL, GraphQL, Socket io, etc.

Have you contributed to other open-source projects? If so, which projects and can we see some of your code?

ListenBrainz is the first Open Source project I’ve contributed to and I think It’s a fantastic start to kick-off my open source journey.

Some of the projects that I have worked on are:

Multiplayer-QuizGame

SolSec

Practical Requirements

I currently use a M1 Macbook Air, 8GB RAM, 256GB SSD.

How much time are you available weekly, and how would you plan to use it?

In the summer break, I can easily dedicate 30-40 hours a week for the project as I won’t have any other engagements.

3 Likes

Hi @rana_satyaraj , thanks for the proposal!

I’m only commenting on the front-end part of the proposal:
Overall the suggested implementation for the BrainzPlayer component looks pretty good!

This really deserves more work after having a good look at the BrainzPlayer functionality: how do you detect the end of a track? How do you update the current seek time on the progress bar, or seek to a specific time ? How do you update the state with the current track’s title and artist, duration? etc.
General waving away of the details doesn’t look great on the proposal :wink:

Keep up the good work !

Hey @mr_monkey , Sure thing, I’ll update the proposal on those details ASAP.

1 Like

Hi!

Thanks for the proposal.

You need to specify the project size of your project as well. Other feedback below:

I think we would like to support setting up multiple subsonic servers per user so the UI will need to be thought better. We can work with aerzol to create mockups for the same. If you want to do a 175 hours project, we can create the UI for linking subsonic servers. If you want to do a 350 hours project, you will need to do implement the UI (mockups will be provided).

I don’t see a subsonic key in Subsonic Auth Docs, it says you need to have a salt but that is randomly generated each time.

We want to support connecting multiple subsonic servers so this won’t work as is. What can be done instead is that the service type column becomes a string column and there is another table or a JSON column to store details of the subsonic server.

The model here is incomplete, you need to take a look at the admin/sql/create_tables.sql in the codebase for the current table definition. username should go in external_service_id field, password can go in access_token. We’ll need to figure out a way to stash the host url though. Could be a new column, or might even be a new table for subsonic servers. Will need to think more.

I don’t think its a good idea to send the password to the frontend, it might be better to create an endpoint on LB backend that frontend can call. Then LB backend calls subsonic with the password and other data. But I see this is the salted password so it might be okay-ish, will need to check.

I think you can do this in 2 weeks max, add another week for UI stuff because it will more involved than proposed in the proposal.

Hey @lucifer

Yes, we can do that, but then we’ll have to store the salt too in the external_services_oauth table and update it accordingly.

Yes, I thought of this, but still went with new fields in the proposal. But sure, we can use those fields to store the username and password, and add one more column for the host URL.
That being said, creating a new table for subsonic servers is a great idea too, especially when we need to store the credentials for multiple subsonic servers.We can discuss it in detail and finalize in the first week.

I don’t think that’s possible, because the backend can’t send requests to the server running locally on users’ machine. And as you said, we are sending the hash of the password, so shouldn’t be a problem.

Sure thing, I’ll make the changes in the timeline.

salts are generated randomly for each request so i don’t think you need to store the salt.

Okay, lets mention this in the proposal for now. Maybe we’ll end up adding both ways, a separate way to authenticate with localhost servers and a separate way to authenticate with internet connected servers.

But we need to include the salt in the link too with the API request

Hey, @lucifer @mr_monkey. Just an update.
I thought I’d give a little clarification, my inactivity in the past month was mainly due to my university final exams (which are set to conclude on May 4th), I hope it doesn’t reflect negatively on my commitment to the project. Thank you guys, for your understanding and continued support.