Recognition quality and languages ​

What Flat's Optical Music Recognition reads well, what it does with lyrics, and what to do when a result is not what you expected.

What OMR reads ​

What kind of sheet music works well? ​

OMR works best on clean, printed sheet music in standard notation: engraved scores, good scans, and sharp photos, from single instruments through grand-staff piano to full orchestral scores. Handwritten music and guitar tablature are not supported. As a rule of thumb, if a page is hard for a person to read, it is hard for Flat to read.

The help center is the source of truth for what is recognized, and it is kept current as the engine improves:

Does score length or complexity affect quality? ​

No. The processing pipeline is the same regardless of how long a score is, and there is no score splitting that would degrade longer pieces. If quality drops partway through a long score, that is a notation or scan issue on those pages, not a length effect. It is worth reporting.

How do I find out about recognition improvements? ​

The OMR changelog covers ML model updates and other significant recognition changes. It does not list every bug fix.

For changes to the API surface itself, endpoints, fields, and behavior, see the REST API changelog.

Languages and lyrics ​

Which languages are supported for lyrics? ​

The supported set is returned as locales (BCP 47 codes) and localesDetails (the same codes with English display names, ready to drop into a language picker) by GET /omr/capabilities, which needs no authentication.

65 languages are currently supported:

Afrikaans (af), Albanian (sq), Arabic (ar), Azerbaijani (az), Bengali (bn), Bosnian (bs), Bulgarian (bg), Chinese (Hong Kong) (zh-HK), Chinese (Simplified) (zh-Hans), Chinese (Taiwan) (zh-TW), Croatian (hr), Czech (cs), Danish (da), Dutch (nl), English (UK) (en-GB), English (US) (en), Estonian (et), Filipino (fil), Finnish (fi), French (fr), French (Canada) (fr-CA), German (de), Greek (el), Hindi (hi), Hungarian (hu), Icelandic (is), Indonesian (id), Irish (ga), Italian (it), Japanese (ja), Japanese (Hiragana) (ja-HIRA), Kannada (kn), Korean (ko), Kurdish (ku), Latin (la), Latvian (lv), Lithuanian (lt), Malay (ms), Maltese (mt), Māori (mi), Marathi (mr), Nepali (ne), Norwegian (Bokmål) (nb), Occitan (oc), Persian (fa), Polish (pl), Portuguese (pt), Portuguese (BR) (pt-BR), Romanian (ro), Russian (ru), Serbian (sr), Slovak (sk), Slovenian (sl), Spanish (es), Swahili (sw), Swedish (sv), Tamil (ta), Telugu (te), Thai (th), Turkish (tr), Ukrainian (uk), Urdu (ur), Uzbek (uz), Vietnamese (vi), Welsh (cy).

New languages are added over time, so build your picker from localesDetails rather than hardcoding a list.

What do locales and mainLanguage actually do? ​

They tell the recognition pipeline which language to read lyrics and text in. They do not affect the notation itself.

They are not just a hint. Flat runs different OCR models depending on the languages you declare, and can run several models over the same file to get the best reading, then links the recognized text back to the note positions. So locales genuinely changes which models see your pages, which is why an accurate value is worth setting and why a wrong one degrades the lyrics.

  • locales is set when you create the job, as an array of BCP 47 codes. Pass the languages the score is likely to use. Defaulting to the user's own locale, when it appears in locales, is a good starting point.
  • mainLanguage is set when you submit the details step and overrides the job locale for the whole score. It is the natural place to let a user correct the language after seeing the detected title.

What happens if the lyrics are in an unsupported language? ​

The job succeeds and the lyrics come back wrong, rather than failing with an error. Characters and diacritics can be mangled, and there is no validation warning to tell you why.

So check that the language you need appears in locales before you build on it. If it is missing, email developers@flat.io: languages get added on request, and several in the list above arrived that way.

Capturing on mobile ​

My users photograph sheet music with a phone. How should I capture it? ​

Present the platform's own on-device document scanner rather than a raw camera shot.

A handheld photo does work. Flat pre-processes what you send, and a reasonably framed photo of a page converts fine. But pre-processing can only recover so much from perspective skew, uneven lighting, a shadow across the staff, or the desk still in frame, and a better input always gives a better result. Sending a scanner-corrected page is the cheapest quality win available on mobile, and it costs you nothing: the correction happens on the phone, before the upload.

Both scanners run entirely on device, are free, add no per-image cost, and give the user edge detection, perspective correction, shading cleanup, multi-page capture, and a review pass before anything is uploaded or a credit is spent. This is what the Flat and Opuscan mobile apps use, and neither app implements its own camera.

PlatformAPIAvailable sinceDependency
iOSVNDocumentCameraViewController (VisionKit)iOS 13None, it is a system framework
AndroidML Kit Document Scanner (GmsDocumentScanning)API 21com.google.android.gms:play-services-mlkit-document-scanner, delivered through Google Play services

Minimal examples, in both cases handing you the captured pages in the order the user confirmed:

swift
import VisionKit

// Presenting the scanner.
let scanner = VNDocumentCameraViewController()
scanner.delegate = self
present(scanner, animated: true)

// VNDocumentCameraViewControllerDelegate.
func documentCameraViewController(
    _ controller: VNDocumentCameraViewController,
    didFinishWith scan: VNDocumentCameraScan
) {
    controller.dismiss(animated: true)
    // VisionKit has no page cap of its own: check the count against maxPages yourself.
    // Pages come back as UIImage, so encode each one before uploading.
    let pages = (0..<scan.pageCount).compactMap {
        scan.imageOfPage(at: $0).jpegData(compressionQuality: 0.8)
    }
    // Upload one page per call, in this order.
}
kotlin
val options = GmsDocumentScannerOptions.Builder()
    .setScannerMode(SCANNER_MODE_BASE)
    .setPageLimit(maxPages)              // from GET /omr/capabilities
    .setResultFormats(RESULT_FORMAT_JPEG)
    .build()

GmsDocumentScanning.getClient(options)
    .getStartScanIntent(activity)
    .addOnSuccessListener { /* launch it */ }
    .addOnFailureListener { /* fall back to the file picker */ }

// In the activity result. RESULT_FORMAT_JPEG means the pages are already
// encoded JPEGs on disk, so there is nothing to convert.
val pages = GmsDocumentScanningResult.fromActivityResultIntent(data)?.pages
val uris = pages?.map { it.imageUri }
// Upload one page per call, in this order.

Both are documented by the platform vendors:

Four things worth knowing before you wire this up:

  • Drive the page limit from the API. Read maxPages from GET /omr/capabilities rather than hardcoding a number, so a user cannot capture more pages than a job will accept, and so the value tracks the limit when it changes. ML Kit enforces it during capture through setPageLimit; VisionKit has no equivalent, so check scan.pageCount once the user is done and send them back if there are too many.
  • Keep a fallback on Android. ML Kit's scanner is delivered through Google Play services, so it is unavailable on devices without them. That surfaces as a failure on getStartScanIntent, not as a missing class, so handle it and fall back to the file picker.
  • The camera permission works differently on each platform. iOS needs an NSCameraUsageDescription string in your Info.plist. On Android the scanner runs inside Play services and owns the camera prompt itself, so your app may need no CAMERA permission at all.
  • VNDocumentCameraViewController does not run on the iOS simulator. Fall back to a photo picker there so the flow stays testable during development.

There is no browser equivalent, so a web upload path still needs a plain file picker.

Should I upload one file per page? ​

The thing to know first: page order is upload order.POST /omr/jobs/{job}/files has no reorder parameter, so the sequence of your calls is the page sequence of the score.

That makes the scanner's own review UI, where the user reorders, rotates, and deletes pages, the step that decides page order. Upload in the order the user confirmed there and you need no reorder screen of your own. Flat's Android app ships none for the camera path for exactly this reason.

So use the Interactive Jobs API for camera capture, and send one file per page. It accepts images directly, so the scanner's pages go straight through with no merging step at all: encode each one as JPEG or PNG and upload it. It is also the flow that gives you the review step, live progress, and MusicXML without a Library score. See Which flow should I use?

If your app also lets people pick an existing photo instead of scanning, note that HEIC and HEIF are accepted alongside JPEG and PNG, so an iPhone photo can be uploaded as-is without transcoding.

How should I size and compress captured pages? ​

Start by uploading the scanner's output unchanged. It is already cropped and corrected, and it is normally well inside the 25 MiB per-file limit (maxFileSize). Flat's Android app uploads ML Kit's JPEGs byte for byte.

Resize only if you need to, and do not be aggressive about it. OMR reads fine detail: ledger lines, articulations, accidentals, and lyric text. Where Flat's apps do resize, they cap the width at 2000 px and re-encode at JPEG quality 0.75 to 0.85, which lands a portrait page around 0.5 to 1 MB. Treat that as a reasonable starting point rather than a Flat requirement.

The counterweight is the same rule that governs the rest of OMR: if a page is hard for a person to read, it is hard for Flat to read. Stop compressing well before the staff lines start to break up.

One scanner setting belongs here too. Flat uses ML Kit's SCANNER_MODE_BASE, notSCANNER_MODE_BASE_WITH_FILTER. The filter modes apply their own image enhancement, which is tuned for text documents rather than for notation.

When a result is wrong ​

A conversion came back inaccurate. What do I do? ​

Email developers@flat.io with the job ID (the id from POST /omr/jobs, or the task id for the auto simple import). That is the fastest thing to give us: it lets us pull up the exact run, its inputs, and its internal quality report. Attaching the source file and a description of what is wrong, ideally with measure numbers, helps too.

Concrete examples are far more useful than a general report, and they are how the engine gets better.

What happens next:

  1. We investigate whether the example is an isolated case or a recurring limitation in the recognition or score-assembly pipeline.
  2. Recurring limitations are fixed in the product.
  3. Once an improvement ships, we can reprocess affected conversions.

There is no automatic credit refund for a conversion that completes but is inaccurate. Credits are reverted for jobs that fail, see Do failed jobs consume credits? If you are running a large batch, tell us before you start rather than after.

How do I know which OMR ML model version produced a result? ​

The version of the OMR machine-learning model that read your pages is embedded in the MusicXML output. Record it alongside the result and you can tell later which of your conversions predate a given improvement, which is exactly what you need when deciding what to reprocess.

Is there a confidence score for a whole job? ​

Not for the job as a whole. What you do get is per-part confidence during the optional details review step: each detected instrument carries a resolvedConfidence of high, medium, or low, which is what the Flat and Opuscan apps use to decide what to put in front of a user for confirmation. Rendering that step is the most effective quality control you can add to an integration.

Copyright © Tutteo Limited

All rights reserved