allowing lilypond and MusicXML source upload

[kcleung]

If the user typeset a work and uploaded its PDF, it would also be a very good idea to upload the lilypond / MusicXML source (an *open-standard* typesetting language) so that files can be modified / recompiled in the future as needed (e.g. produce different transpositions or printing in braille etc)

However if each source file is uploaded individually, this would certainly clutter the page very quickly because in Lilypond, a file may include other files (like source code in a computer program) and each PDF file may be generated from multiple source file (a source file may also contribute to multiple PDF files.

For typeset score (with *open* format - Lilypond and MusicXML only), we should set up a compiler on the server which automatically reads and unzips a zip file and compile PDF from the source file (like what Mutopia does). We can also stipulate a specific file structure for the zip file and the compile script (makefile). After specifying format of the source file on the submission screen, the server automatically opens the zip file and runs the makefile. If compile fails or if zip file contains non-text files, reject immediately.

Then the human examiner will examine the PDF output as usual. This would allow source to be uploaded yet will not incur extra human overhead.

Proprietary formats (such as overture, sib, finale) should remain unaccepted because they have closed format, and they can be converted to the *open* format MusicXML.

In this way, the source file can be adapted by others for their own need with minimal cost.

[Yagan Kiely]

Correct me if I'm wrong, but lilypond format can't be opened by any of the large notation programs?

Musicxml (as a universal standard (or getting there)) is an option, but not notation program specific files - regardless of open source.

The reason why sib and finale files should not be accepted is not because they are commercial, but because they are not easily accessible by all. XML, is slightly better as more notation softwares are using it.

[ras1]

.ly is only readable by Lilypond, but it might not hurt to have it in addition to the pdf. Ease of transposition, editing, correction, etc.

[horndude77]

This discussion comes up from time to time. It's true re-typesets don't add much value here unless the source file is available (for many of the reasons you mentioned) or the original scan was extremely low quality. In my mind, the purpose of IMSLP is for scans first. The lilypond file format is too volatile to really be suited for archival purposes. For these reasons I don't upload my typesettings here (and because I still consider them still in need of much polish and review). Mutopia goes to great lengths to be able to compile files made with old versions of lilypond. (The really old ones we just have to manually update). I agree with Yagan that MusicXML is the best option of the notation formats if we allow any. Mutopia is good for lilypond files instead.

[kcleung]

But the Lilypond compiler is open source and available to any systems (just download it)

http://www.lilypond.org

In terms of functionality, Lilypond is far superior than MusicXML. Also Lilypond is much more human-readable and editable than MusicXML. The functionality of MusicXML is quite limited.

Mutopia is good for Lilypond, but its admission criteria are much stricter than IMSLP. A document must be published before 1923, PD in US, Commnonwealth and Europe before it can be uploaded. This would exclude most early Chinese Symphonic works which I intend to re-tyset (e.g. works by Xian Xinghai) to get parts score and transposition.

For processing Lilypond zip files, why can't we just lift the system setup off Lilypond (for compiling and setup? I am sure they only use open source software / script which we can use directly.

Lilypond format is backward compartible, as a new version of lilypond come out, you may want to maintain the code to make use of the newest feature. But it is not compulsary. It will still compile by future lilypond compilers if it can compile now. Moreover we still store the initially-compiled PDF file.

Therefore in terms for long-term archival, Lilypond is still suitable. Of course the generated PDF file is also part of our archive.

Moreover, think about the visually-impaired musicians! A MusicXML file or a Lilupond file would potentially allow a compiler with Braille plugin to generate Braille files in a fly (rather than retysetting in Braille)

So I strongly believe that we should set up Lilypond and MusicXML submission and compiiling system on our website.

Yagan: I am busy for the next two months (my PhD work). But after that I would like to work closely on integrating these services into IMSLP. Moreover we can enlist help from the Mutopia technical team. I am sure they will be more than happy to help us.

[daphnis]

I disagree with submission of typeset files on the grounds of consistency. All users, regardless of what program they use to typeset a work, can freely generate a PDF. This is what we will gladly host, and if some user would like the source typeset file, all they have to do is request it from the user (since we assume users submitting typesettings are the creators). There are other sites as previously mentioned that only accept typesettings. Let those sites be the repositories for such files. We will continue to accept only PDF files as a universal format for the storage and dissemination of sheet music as this is the format which bests supports our objective in retaining both old engraved editions and new electronic typesets. We no longer accept ZIP archives or dejavu and probably will not in the future.

[pml]

Dear KC Leung,

viewtopic.php?p=7874

Already answered, at length...

IMSLP isn't pushing formats that aren't supported widely, and actually, both the Sibelius and Finale formats are a good deal more tested and mature than either of Lilypond or MusicXML. The latter is probably the most flexible for importing and exporting into typesetting software, but it is still much less supported than MIDI - which is generally inadequate for music notation (if not an actual performance).

Regards, Philip

[kcleung]

There are other sites as previously mentioned that only accept typesettings. Let those sites be the repositories for such files.

But the problem is that license of these other sites (such as mutopia) has a much stricter copyright clearance requirements than IMSLP. (must be simutaneously PD in USA, Commonwealth and EU)

What we need is a site which uses the Mutopia setup, but with IMSLP rules and host in a life + 50years country (like Canada). In this site, it first starts by importing / linking all mutopia entries and possibly entries from other PD MusicXML sites, then it builds up from there.

Yagan: This project would really help me (and others) to retypeset many rare Chinese Symphonic scores and spits out whatever fullscore, parts, parts with special transposition, braille parts etc. Do you know anyone who would be interested in a Mutopia-like PD50+ typeset repository for Lilypond and MusicXML?

[pml]

Hi KCL,

Without an open source music OCR package, your best option is still to make PDFs of rare Chinese full scores, and allow end-users to typeset parts etc. as needed/on demand/on request. Collating the derivatives from a full score is generally a much lengthier and demanding task if you are wishing them to be both accurate and usable; but if an orchestra is attempting one of these pieces, it is not too much work for someone to do several months in advance provided the full score is available. In that case you would best structure a works page like:

== Full score ==
1. Full score (PDF) <-- this to be added initially

== Instrumental Parts ==
2.1 Violin 1 (PDF)
2.2 Violin 2 (PDF)
...
2.x Last part of "x" parts (PDF)
2.x+1 Archive of source files for parts AND full score (ZIP) <-- these to be provided eventually

No matter which typesetting package is used, it is unlikely to meet the needs of any possible end-user, so the compromise would be to use PDFs as the default and the actual source files or a generic MusicXML conversion, archived in a standard compression format.

Regards, PML

[Yagan Kiely]

Everything has been said, but:

Typesets are not something that IMSLP wants to promote. Most unprofessional typesets (which is what these are) are far from anything resembling publishable standards - they are ugly.

For processing Lilypond zip files, why can't we just lift the system setup off Lilypond (for compiling and setup? I am sure they only use open source software / script which we can use directly.

If *.ly, then we also need the other two, or three, or four 'major' formats as they are much more widely used. Yes IMSLP would like to support open source as much as possible, but only if it doesn't effect user experience or ease of maintaining.

Moreover, think about the visually-impaired musicians! A MusicXML file or a Lilupond file would potentially allow a compiler with Braille plugin to generate Braille files in a fly (rather than retysetting in Braille)

This also isn't much of a concern as the percentage of visually-impaired musicians accessing IMSLP will be extremely slim.

Yagan: I am busy for the next two months (my PhD work). But after that I would like to work closely on integrating these services into IMSLP. Moreover we can enlist help from the Mutopia technical team. I am sure they will be more than happy to help us.

I am against the idea of having any format other than PDF (unless a new standard comes out). I won't to do the OM project but I do not want (nor can) help with this.

[daphnis]

Once again, this is why PDF is the universal standard it is, and why we choose to support only that. If people are interested in obtaining the "source", then let them inquiry directly to that source.

[ras1]

Yagan: While I'm fine with only accepting PDFs, I also think there's nothing wrong with typesets. Yes, they're not ideal, and certainly inferior to a published score, but much better, in my opinion, than no score at all.

[kcleung]

PML:

Thanks for clarifying the workflow and this was exactly want I was doing with the Chinese Rhapsody. I am a full-time PhD student and of course have no time typesetting the whole score!!! This is why I first scanned and uploaded the *full score* and if some orchestras are interested in this work, *they* can do whatever they like (such as retypesetting into Lilypond etc. and upload the PDF parts onto IMSLP.

Yagan and PML:

However after retypesetting the work, it would be in everyone's interest to make it available so that it can be easily adapted for other previously mentioned purposes (this is one of the motivation for typesetting). Since IMSLP is a PDF-only site, one would need a Lilypond/MusicXML site (can be independent from IMSLP) that:

- automatically accepts items form Mutopia
- only accept other entries which PDF version already accepted by IMSLP (together with the copyright classification assigned by IMSLP)
- should an item be blocked / pulled in Mutopia / IMSLP, this item should be automatically blocked from the source site

So the orchestra mentioned above still need to upload all derived PDF to IMSLP, then it can upload the source to the source site (indep from IMSLP) as a zip file and in the IMSLP work page make a link to the entry in the source site.

In this way we can make IMSLP consistent while still allowing others to see the source.

Yagan:

Actaully re-typesetting is a last resort, although I also learn lilypond as a trade-skill. The motivation for me (and you) to get involved in the OM project is to *avoid* the need for re-typesetting in 99.9999999% of cases (Western symphonic works)!!!!!! This is why I am so keen to see the implementation of the necessary infrastructure to improve the usability of the orchestral score. The priority of the OM project is of course a magnitude higher than the source-loading project.

[Yagan Kiely]

Yagan: While I'm fine with only accepting PDFs, I also think there's nothing wrong with typesets. Yes, they're not ideal, and certainly inferior to a published score, but much better, in my opinion, than no score at all.

Yes! It is only my opinion also. That said,
I don't think IMSLP should allow bad typesets, as it damages IMSLP's reputation.

[kcleung]

Yagan: While I'm fine with only accepting PDFs, I also think there's nothing wrong with typesets. Yes, they're not ideal, and certainly inferior to a published score, but much better, in my opinion, than no score at all.

Yes! It is only my opinion also. That said,
I don't think IMSLP should allow bad typesets, as it damages IMSLP's reputation.

Well, this is already enforced. If a reviewer thinks the typeset is unacceptable, the submission will be rejected (through our current "scan quality" rating system).

Under the current system, a typesetter would get the generated PDF submitted and accepted into IMSLP, then he puts his source files onto an external website and make a link on the IMSLP page pointing to the file on that site.

However what will be the difference if we have a FTP server for storing source zip for accepted typeset PDF files and the "external link" points to the zip stored on the FTP server? The "external link" just points to the FTP server instead of the user's homepage etc. This way shouldn't increase our administrative load or change the way we work, yet allowing a secure way to make the source available. As the server consists of zipped ASCII files, they shouldn't take a lot of space (maybe 1-2GB would be plenty for the server) and also there aren't that many re-typesets on our site at the moment). If you worry about malware etc, the server can easily screen all zip files to ensure they are not encrypted and they only contain text files.