This was a tool my old company Topologi made for JSTOR, in part as an R&D project to explore the viability of the area (and becase JSTOR needed it).
So it was strictly limited to only implementing the things that JSTOR needed, and we also decided to only use XSLT 2 for it.
The takeaways I got from it were:
1) The approach works, but it would require a deal more work (and testing) to be complete and correct.
2) XSD has many more edge cases than even I expected. One of the our developers working on it subequently got a PhD in software, and he really struggled with it.
3) It is difficult to structure and handle so many steps in a pipeline with XSLT. There were lots of parts where a language with mutable tree would have been much more straightforward.
4) Schematron is fine for validating datatypes and derived types.
5) Schematron is fine for validating keys/IDs etc.
6) Schematron is fine for validating content model with no repeated .element names.
7) Schematron can handle validating content models with the same element names in more than one position in a content model using regular expression string matching, but this gives less useful messages.
8) A complex content model was actually often the sign that grammars were not elegant to express the constraints. Sometimes handwritten assertions could do a much better job at capturing the real constraint, rather than a verbose Schematron version of a verbose Content Model version of the constraints!
I cannot remember which parts we left unimplemented: perhaps blocking and substitution groups? Anyway, try it and see if it handles your cases. That code is also looking for a maintainer, if you want a nice little project.