Blockaoke

At Blockaoke, we mix blockchain with music. Let AI decide who is the best at singing in a karaoke battle with your friends along with an enriching web3 experience.

Once you enter the app, choose your NFT avatar and sign in with your wallet in the lobby to enter the karaoke room. Once all your friends are in the room, choose a song you want to sing. The lyrics will be displayed and the karaoke background music will begin playing. You can pass on to the next user to sing and when everyone has sung their heart out in the karaoke competition, we will display the scores and winner. You can store your recordings on IPFS for memories in the future.

In the future, we will add token gated meetings on our custom URL and make it an entertainment platform, which can include 'who is the best dancer or comedian' along with karaoke. We would also customize our subdomain UI.

We used Huddle01 iframe and ReactJS SDK for an awesome frontend and smooth user experience! It helped us integrate features such as wallet sign in, NFT avatars, meeting recording and storing on IPFS. Cross chain audio and video communication is an integral part of our app and Huddle01 helps us achieve that. Huddle01 mentors helped us guide during crucial moments of the hackathon so a big shoutout to them. Further, we used React Bootstrap for styling and bundled everything in a NextJS app. We linked this to our ML model that is made in Python on Google Colab using Tensorflow and libraries like Librosa, Moviepy and Matplotlib.

Also, current karaoke systems provide scores only on the basis of lyrics sung, but we use tone and rhythm to decide whose song best matches the original song. We calculate the scores using a siamese neural network. First, we convert the audio files of each singer to a mel spectrogram, which is an image representation of the audio on a frequency time graph where colour shades represent amplitudes. We trained our ML model with half positive samples (spectrograms of good covers of a song) and 5 negative samples. We wrote a contrastive loss function to optimize our parameters and improve accuracy. The main task of the neural network is to calculate euclidean distance. The lower this distance, the better the singer's audio matches original song. When normalized, we give a score from 0 to 100 to users.