GSOC 2022: Integrate Apple Music services for recording listens and playing music

Name: Xianyi Nie

IRC: Xianyi

Github: Xianyi Nie

Email: niexianyi01@gmail.com

Timezone: GMT-4 hours

Integrate more music services for recording listens and playing music

Project Overview

ListenBrainz has a number of music discovery features that use BrainzPlayer to facilitate track playback.BrainzPlayer (BP) is a custom React component in ListenBrainz that uses multiple data sources to search and play a track. As of now, it supports Spotify, Youtube and Soundcloud as a backend. ListenBrainz zalso supports linking a Spotify account to record listening history. Integrate the external music service(Apple Music) in ListenBrainz to make adding other music services easier.

Implementation

1. Allow the user to link the Apple Music to record history

1.1 Modify the database and add functions which can get data we need through the sqlalchemy

  • Modify the USER ListenBrainz database, for example: I would modify the admin/sql/create_types.sql, add new type

    ALTER TYPE external_service_oauth_type ADD VALUE 'appleMusic';
    
  • Add the function which can returns a list of users whose listens should be imported from Apple Music;

    def get_active_users_to_process() -> List[dict]:
    
  • Add the function which can return user’s Apple Music linking details to display on connect services page;

    def get_user_import_details(user_id: int) -> Optional[dict]:
    
  • Add the function get the recent listened list of the Apple Music;

    def get_user(user_id: int) -> Optional[dict]:
    

1.2 Set up the Apple Music API configuration and save the user Oauth token to the Backend for storage

  • Generating Developer Tokens
  • Authorize users to connect the AppleMusic service through the Apple Music Documentation API:
    const appleMusic = MusicKit.getInstance();
    
    // Check authorization before accessing user's iCloud Music Library:
    await appleMusic.authorize();
    const result = await appleMusic.api.request('v1/me/library/played');
    // User's iCloud Music History
    console.log(result.data.data); //test the data
    
  • Backend: Add the following functions which can save the token in the database
    def get_user(self, user_id: int) -> Optional[dict]:
           # get the message of the user
          return applemusic.get_user(user_id)
      
    def add_new_user(self, user_id: int, token: dict) -> bool:
        external_service_oauth.save_token(user_id=user_id, service=self.service, access_token=access_token,
                                                  refresh_token=refresh_token, token_expires_ts=expires_at,
                                                  record_listens=active, scopes=scopes)
    
  • WebServer: Set an Endpoint to accept the setting from the user and delivered the user message
    @profile_bp.route('/music-services/details/', methods=['GET'])
    @login_required
    def music_services_details():
        applemusic_service = AppleMusicService()
        applemusic_user = applemusic_service.get_user(current_user.id)
      // Implement the Apple Music Permission here
        return render_template(
            'user/music_services.html',
            applemusic_user=applemusic_user,
            current_appleMusic_permissions=current_applemusic_permissions,
        )
    

1.3 Deal with the user authorization process

  • Backend: Add the following functions which can get the authorization information:

    def get_authorize_url(self, permissions: Sequence[str]):
    
    def fetch_access_token(self, code: str):
        """ Get a valid Apple Music Access token given the code.
        Returns JS:
            a dict with the following keys     
               {
                'access_token',
                'token_type',
                'scope',
                'expires_in',
                'refresh_token',
                }
        """
            
             
    
  • Webserver: Set an Endpoint for the music service callback to finish the authorization process

    @profile_bp.route('/music-services/<service_name>/refresh/', methods=['POST'])
    @api_login_required
    def refresh_service_token(service_name: str):
       """ Returns JS:
          a dict with the following keys
          {
             'access_token'
          }
    

1.4 Deal with the process of refreshing the access token

  • Backend: Add the following functions which can revoke the new issued token and ask the user to authenticate again

      def refresh_access_token(self, user_id: int, refresh_token: str):
          """ Refreshes the user token for the given applemusic user.
          Returns:
              user (dict): the same user with updated tokens
    
  • Webserver: Set an Endpoint to deliver the new access token

    @profile_bp.route('/music-services/<service_name>/refresh/', methods=['POST'])
    @api_login_required
    def refresh_service_token(service_name: str):
       """ Returns JS:
          a dict with the following keys
          {
             'access_token'
          }
    

1.5 Deal with the process of revoking the access token

  • Backend: Add the following functions which can revoke the new issued token and ask the user to authenticate again

    def revoke_user(self, user_id: int):
    
    def _revoke_token(self, user_id: int, access_token: str):
       """ Returns JS:
          a dict with the following keys
          {
             'access_token'
          }
    
  • Webserver: In order to allow user changing permissions in a single step, we need to set an Endpoint to disconnect the services.

    @profile_bp.route('/music-services/<service_name>/disconnect/', methods=['POST'])
    @api_login_required
    def music_services_disconnect(service_name: str):
    

1.6 Add front-end components to allow users to link the Apple Music

  • Noted: This is the demo of the part HTML

    <div class="music-service-selection">
        <form action="{{ url_for('profile.music_services_disconnect', service_name='applemusic') }}"
              method="post">
            {{ service_permission_button("applemusic", current_applemusic_permissions, "both",
            "Activate both features (Recommended)",
            "We will record your listening history permanently and make it available for others to view and explore. "
            "Discover and play songs directly on ListenBrainz, and import/export your playlist to and from Apple Music.") }}
    
    
  • The recent listens history can be displayed in the list

1.7 Continue updating recently played resources from Apple Music API and submit them to ListenBrainz

  • Set the update interval

        Update_Interval = 60
    
  • Add the function which can make an request to the Apple Music API for particular user at specified endpoint

     def make_api_request(user: dict, endpoint: str):
    
    • Request URL:

      https://api.music.apple.com/v1/me/recent/played
      
    • Query Parameters:

      Parameters Type Content
      l String The localization to use, specified by a language tag
      limit Integer The number of objects or number of objects in the specified relationship returned.
      offset String The offset to use for a paginated request.
      include String Additional relationships to include in the fetch.
      extend String A list of attribute extensions to apply to resources in the response.
  • Add the function which can update recently played songs for this user and submit them to ListenBrainz, set the update interval.

     def process_user(user: dict, service: SpotifyService) -> int:
    
     def submit_listens_to_listenbrainz(user: Dict, listens: List, listen_type=LISTEN_TYPE_IMPORT):
    
    • Main response properties of the resource:
      Parameters Type Content
      id String (Required) Persistent identifier of the resource.
      type String (Required) The type of resource.
      href String A URL subpath that fetches the resource as the primary object. This member is only present in responses.
      meta Resource.Meta Information about the request or response. The members may be any of the endpoint parameters.

2. Integrate the AppleMusic in BrainzPlayer

  • The front-end page of the functional mock-ups is shown in the following figure:

2.1 Add the AppleMusicPlayer Interface in BrainzPlayer, for example:

export default class AppleMusicPlayer
  extends React.Component<AppleMusicPlayerProps, AppleMusicPlayerState>
  implements DataSourceType{

}
  • Implement the constructor in export default class;
  • Add the function that can get the trackID from the listen;
  • Add the function that can get the link from the listen, etc;

2.2 Implement RefObject of React for Apple Music

  • Update the AppleMusic Player in different functions.
  • Update the render with AppleMusic Player.

2.3 Add front-end components to allow users to link the Apple Music:

image

TimeLine

Before the GSoC (Before 19th May)

  • Set up ListenBrainz local Server development environment
  • Read the related tickets and solve the good-first-bug.
  • Discuss with the mentor and community members about my project and ask for suggestions.

Community Bonding Period

  • Discuss more details with my mentor and community members, bonding with other developer.
  • Fix more issues in Jira related to ListenBrainz and read more documents to understand the project more better.

Phase 1: Week 1 - Week 6

Week 1

  • I will modify the database and finish the class which can get data we need through the sqlalchemy
  • Get familiar with the class in the whole project.

Week 2

  • I will set up the AppleMusic API configuration and save the user Oauth token to the Backend for storage

Week 3

  • I will complete the process of the authorization information.
  • Write document about the current part.

Week 4

  • I will deal with the process of refreshing and revoking access token. and finish both of the Backend part and the Webserver part.

Week 5

  • I will finish the class can continue updating recently played resources from Apple Music API and submit them to ListenBrainz

Week 6

  • I will add front-end components and test the part of the link works fine.
  • Write document about the current part.

Milestone: Preliminary Deliverables

  • Support linking a Apple Music account to record listening history and display them in the website.

Phase 2: Week 7 - Week 12

Week 7-8

  • I will write JSX to add the AppleMusicPlayer Interface in BrainzPlayer.
  • Integrate the AppleMusic Player as a RefObject of React in BrainzPlayer and work on the render part.

Week 9-10

  • I will update the front-end components of the BrainzPlayer and discuss with my mentor and community members about the suggestions of my current part.
  • Write document about the current part.

Week 11

  • Fix related bugs, and write and combine the document of the whole part.

Week 12

  • Buffer for unexpected delay and issues, and discuss with my mentor and community members about the suggestions of my final evaluation deliverables

Final Evaluation Deliverables

  • A new music service with AppleMusic integration for users to play and record listens on ListenBrainz.

After the GSoC

  • I would continue contributing to ListenBrainz in the future and add more future of the BrainzPlayer
  • Try to become a mentor at the next GSoC.

Information about Myself

My name is Xianyi Nie, I’m currently a graduate student at Georgia Institute of Technology and my major is Computer science & Engineering. This is the first time I’m participating in GSoC and I’m so excited. I am new to open source contributions, but I am well aware of Java,Python, data structures and algorithm, HTML, CSS, and JavaScript, SQL, React. Besides, I have experience in full-stack development.

Tell us about the computer(s) you have available for working on your SoC project!

I have a MacBookPro with the M1 chip and 16 GB RAM

When did you first start programming ?

I began to program in C language in the fourth grade of my primary school

What type of music do you listen to ?

I usually listen to pop from Taylor Alison Swift , Jay Chou . Besides, I also listen to the Classical Guitar Music and Classical Piano Music.

What aspects of the project you’re applying for interest you the most?

In fact, I’m a music enthusiast, and I’m also a programming enthusiast. I have played the guitar for more than five years, and my piano skill has reached level 10. Once I found that this project is related to my technology stack, and the ListenBrainz aims to keeps tracks of what music you listen, I can’t wait to devote myself to this project and the community of MetaBrainz. Because I was trying to find an integrated website can collect all the music I loved, instead of being divided in different music app. I really intersted in this ListenBrainz project so I want to make it better through my efforts.

Have you ever used MusicBrainz to tag your files?

I haven’t used it because I didn’t know it before. And now I start to use it.

If you have not contributed to open source projects, do you have other code we can look at?

Yes, I did various full-stack projects like Travel Reservations Service System, Parking Spaces Detection Based on Machine Learning and many other projects which are available on my GitHub.

What sorts of programming projects have you done on your own time?

I have done some full-stack projects on my own time, like Travel Reservations Service System, Customer Feedback-Collection Application.

In the project of Flight Reservations System:

  • Developed a full-stack travel reservation project for users and administrators, which can help users manage travel itineraries in an integrated manner and allow administrators to manage flights and customers.
  • Implemented stored procedures to modify the database state and implemented views to provide the data presentation based on MySQL workbench.
  • Designed the user interfaces by React frameworks in JavaScript to complement front-end web integrated with back-end database system by Express.

In the project of Customer Feedback-Collection Application:

  • Developed a feedback-collection application for start-up company, which can be applied to collect survey responses from apps and generate a visualization report to customers.
  • Applied NodeJS in JavaScript to construct the server on Heroku and build reusable user inputs with Redux Form, completed with navigation, handled credit cards, and received payments from users with Stripe.
  • Utilized Email provider recorded the feedback, stored the collected data in MongoDB by Express backend.
  • Enhanced authentication flows in app with Google OAuth authentication.

How much time do you have available, and how would you plan to use it?

I plan to use 30 hours per week to the project. The total duration of the 12-week project is about 350 hours. And I can contribute more time to the community if the project needed.

5 Likes

Hi! As we talked about last time, I add more detail about the Endpoint after diving into the Apple Music API; Besides, I also update the timeline. Love to hear your suggestions! :smiley:

Hi! Thanks for the proposal. Before reviewing further I wanted to confirm the deliverables for this project. I see that you intend to link the Apple Music Player to BrainzPlayer. You also mention about the submitting listens service part. Do you intend to do both parts (will be 350 hrs project) or only the first (will be a 175 hrs project)?

Thank you so much for your reply! I intend to do both parts.

Thanks. Some comments:

  1. For storing authentication data from Apple Music, we don’t need to create a new table. These existing tables should hopefully suffice. listenbrainz-server/create_tables.sql at ed67b5f851c13fbc88068e4299cec654a3b0f6e8 · metabrainz/listenbrainz-server · GitHub. These tables store tokens from Last.FM, Libre.FM, CB, Spotify. You’ll only to add an enum entry for apple like this: listenbrainz-server/2021-07-10-add-lastfm-external-service-type.sql at master · metabrainz/listenbrainz-server · GitHub and it should be enough.

  2. The Apple MusicKit authorization code will run in the website whereas the functions you mention are part of the backend.
    How will these communicate with each other, for instance how does the client send the Oauth token to the backend for storage, how does the frontend get the necessary client ids if any to initialize MusicKit so on.?
    I suggest to look into how the spotify connections are done for more insight into this. See listenbrainz-server/profile.py at ed67b5f851c13fbc88068e4299cec654a3b0f6e8 · metabrainz/listenbrainz-server · GitHub

  3. Please include what endpoints are needed to be used for refreshing and revoking the access token respectively.

  4. @mr_monkey can review the frontend part but based on current details how will the Open in AppleMusic work button? For instance, Open in Spotify needs a spotify id to open the track on spotify. What ids would Open in AppleMusic need?

Thanks so much for your suggestions!
Yes, I only mentioned how the file in domain works, I will also include the webserver function. And I will update the database part and the token part also.
As for the 4, the response we get from the Apple music contains the url of the music, so I think it also need the AppleId after authorization.

Hi! I updated my PR again and finished the request change, Please let me know if there is any room for more improvements :smiley: