piano2sheet
Table of Contents
Converts Synthesia videos to sheet music.
Demo #
Learning objectives #
The story behind this project is that both myself and a friend were interested in the field of Computer Science; however, neither of us knew the first thing about coding. This was the very first project we worked on to see if we had the motivation to learn a new skill on our own time.
- Attempt to learn a programming language (Python) through online resources
- Share the knowledge we’ve learned and encourage each other
- Wrap up a project together - unfortunately, we got as far as detecting the notes, as seen in the
.gifabove! - Determine whether or not this is a career we could see ourselves pursuing - the answer to this is an emphatic YES
Technologies used #
| Technology | Description |
|---|---|
| Python | used in conjunction with Jupyter to develop the script |
| OpenCV | an open-source library with many computer vision tools that were helpful in detecting the keyboard notes being played |
How does it work #
As MIDI keyboards are capable of exporting key presses to Synthesia in a very systematic manner, we can use this to detect which keys are being played, and when.
1. The video output is consistently ordered
There are 52 white keys and 36 black keys on a standard piano.
2. The key portion can be extracted
We can crop and threshold the image to identify the keys.
3. We can run detection algorithms on the thresholded image
Check out this GIF that highlights the detected black keys. This can be similarly performed for the white keys.
4. From any image, we can perform the same thresholding
Side-by-side comparison of the raw image and thesholded image.
5. Fully connected notes need to be segmented
This image shows two distinctly coloured notes that can be thresholded by using a histogram and identifying the peak.
The result is:
6. And the full detection run on the image
7. Perform this on every frame of the video