How to Integrate NotateMe, the Music Handwriting App, With Your Sibelius Workflow

By in Music Creation

My previous article introduced our NotateMe music handwriting app and how it came to be. This article highlights how to use NotateMe and integrate it into your Sibelius workflow, with a focus on the design decisions that made it revolutionary.

A brief history of notation software will put NotateMe in context, demonstrating the successive technological advancements that have revolutionized the way musicians work with their scores (a detailed account can be found at Music Printing History):

1960s The Musicwriter keyboard created punched cards to be processed by an ILLIAC supercomputer.
1970s The dedicated Musiccomp all-in-one system featured a special keyboard and computer screen.

Musiccomp Score Writer (1977)

1980s MS-DOS based SCORE ran on desktop computers.
1990s Sibelius utilized powerful mouse and windows-based systems. In order to benefit fully from the technology the creators, Ben and Jonathan Finn, designed a novel music notation interface from scratch.
2010s Touchscreen based tablets and phones became sufficiently powerful to run demanding and sophisticated handwriting recognition software.

Martin Dawe Demonstrating Neurotron’s OCR System at Acorn World 1993, London

Designing Something New

At Neuratron we have been developing recognition software since 1993. We knew our work on scanned handwritten music recognition could form the basis of a new type of notation software that took advantage of modern touchscreen technology. For me, it was a rare opportunity to design a user interface from the ground up and create an original product that felt completely intuitive.

NotateMe v1 (2013) with Split Screen Running on iPhone 5

We took the plunge in summer 2012. It was a gamble for us: There are no fixed ways of writing recognition software, and no certainties that it will ever work. In addition there were concerns over whether we could finance its development, as customers’ expectations of what they should pay for apps was too low to be economically viable. We did not even know whether anybody would want to buy a music handwriting app! But I had a hunch, it felt right, and so we went for it full steam.

Whenever I create something original, I like as little disturbance and unnecessary thought processes in my head as possible – I work best when my whole mind is working with a unified objective. I realized this was the key to inventing a music notation app that would give musicians the opportunity to imagine their most brilliant work.

Leonard Bernstein Making Annotations to a Musical Score (click photo for copyright)

I noticed that composers often reach a similar relaxed state of mind when working with pen and paper, in front of their piano or at a desk. It seemed that a portable touchscreen tablet or phone, particularly with a stylus, could become the modern-day equivalent of this whilst offering so much more, including the freedom to write anywhere.

One of the challenges when designing NotateMe was holding on to the simplicity of using pen and paper, while at the same time adding the functionality to edit, print and play back a score automatically generated from handwriting. I felt it important there should be no menus or complicated toolbars in sight, and no keyboard shortcuts to remember.

To maintain the natural pen and paper philosophy, it was also important that musicians could write in the style they were already comfortable with. From a developmental point-of-view, it would have been an easy shortcut to take, to force the user into learning a special way to write notation. Instead we designed NotateMe to quickly adapt to musicians’ handwriting styles, not vice versa.

In the same light, I also thought it important that users should be able to help NotateMe correct any misread symbols by not only erasing and starting over, but by allowing scribbles to be marked more clearly with additional strokes – in the same way people clarify their handwriting using real pen and paper.

Interactive Paper

Armed with the power of a tablet, we added various capabilities to NotateMe that are just not possible with traditional pen and paper:

  • We thought it would be cool and useful for NotateMe to immediately transcribe and play back notes the moment they were written.
  • We rethought the process of selecting and editing objects by making it possible to intuitively draw a lasso around objects to select them. The selection can be tapped and dragged to move it, double-tap dragged to create a copy, and flicked to erase it.

Selecting Musical Objects in NotateMe By Drawing a Lasso

  • We automated the following aspects of score entry to reduce time and tedium from the score-writing process:
    • Voice numbering is not generally marked or highlighted by a musician writing on paper – we thought, why do this on a computer?
    • Tuplets are automatically added once a bar is complete.
    • New bars are added automatically.
    • Clefs, time and key signatures and barlines are all pre-written – simply select and drag to adjust them.
    • Page formatting for saving PDFs or printing.
  • With PhotoScore & NotateMe 8, tap Send to Sibelius and your score will open immediately within Sibelius. Continue with more advanced editing tasks, such as setting the house style, syncing with film, and so on.

Screen Estate

When we began development, we weren’t sure how this could all fit within the confines of phone and tablet displays. Should users enter music onto staves already formatted into pages (as in Sibelius), or long continuous staves that could be navigated by swiping left/right? Was it better to keep the handwriting area of the screen separate from the transcribed notation, or should there be one set of staves for both handwritten and transcribed notation?

With the first incarnation of NotateMe, the answers ultimately arose not only from the reality that tablet and smartphone screens were quite small, but also that they lacked active pens/styluses (meaning the touchscreen could not tell the difference between stylus and finger strokes). A user’s finger/stylus would be required to both write and navigate around the score, and so the NotateMe screen was split in two – the bottom half for handwriting, and the top half for panning.

NotateMe 3 (PhotoScore & NotateMe Ultimate 8) on Microsoft Surface Pro 3

With the release of PhotoScore & NotateMe 8 and its compatibility with larger tablets such as the Microsoft Surface Pro 3 with active Surface Pen, it became possible to use the pen solely for handwriting, and the finger only for navigation.  That is why the latest NotateMe 3 gives users the option to write directly onto the main score. The split-screen option has not been abandoned however. Many users prefer to work directly with their own handwriting; and music educators find the split-screen layout the perfect way to assess and assist students learning the age-old art of music notation writing.

PhotoScore & NotateMe Ultimate 8

Sibelius has shipped with free Lite versions of Neuratron’s music scanning and recognition technology since before even the first Windows version (Optical Manuscript shipped with Sibelius for Acorn computers in 1997). Sibelius 8 is no different and includes PhotoScore Lite 8, which now features the reduced functionality version of our music handwriting app, NotateMe Now.

PhotoScore & NotateMe Ultimate 8 is currently available to purchase at heavily discounted bundle pricing both with new Sibelius 8 purchases and upgrades.

If you already own an earlier version of PhotoScore Ultimate, you can upgrade to version 8 with NotateMe.

Check out the different Sibelius bundles available from the Avid store including the Sibelius + Ultimate Bundle that includes PhotoScore & NotateMe Ultimate 8 and AudioScore Ultimate 8

Get Your Free 30-Day Sibelius Trial

Get Your Free 30-Day Trial

Experience the fastest, smartest, easiest way to write music with Sibelius—the world’s best-selling music notation software.

Neuratron (née Neurotron) was launched from my bedroom when I was 17. Twenty two years on, I’m responsible for a wealth of products including PhotoScore, AudioScore, Hit’n’Mix and NotateMe. I remain in charge of executive and technical decisions at Neuratron, and have a dedicated team working with me.