Please see my comment on the RFC for some background.
My idea of how to keep plugin state and editor state in sync was to use transaction metadata, I see steps as document-changing things and nothing more. This makes them nicely scoped and simple to define.
But I’m aware that integrating such metadata with history and, to a lesser degree, collaborative editing is a huge pain. And treating non-document-changes like steps might be an elegant solution to that. However, I don’t really see how the subclass idea would work—steps are applied one at a time, but states are updated in one go, per transaction. Also, the subclass’ interface wouldn’t be a superset of the Step interface (you can’t apply or invert such steps in the same way as document steps).
For CodeMirror, which we’re rebuilding with a transaction model very similar to ProseMirror, we’re planning to make it possible to store custom items in the history. This hasn’t actually been planned out in detail, so that’s not all that helpful, but the plan would be to have a given transaction metadata slot that the history plugin looks at and, if present, stores its value before the transaction’s steps. When that item is undone, this value is applied in way where it can add to the transaction created for the undo to somehow restore some non-document state.
Such a thing would require a little more code than just having steps automatically picked up, but not that much. Because history isn’t just naive rollback in ProseMirror (and CodeMirror), such code would have to be careful to restore that state without getting out of sync with the document, but that issue would exist with steps too.
But again, as described in the RFC, you can’t just mutate transactions to, for example, automatically add data for deleted ranges. So maybe this is also not the right formulation, and we’d actually need something that allows the history to compute this undoable data when it’s storing a given step, rather than putting it in the transaction somewhere.
(That is, in fact, how docChanged is implemented (return this.steps.length > 0), but yeah, if we remove the assumption that steps apply to the document, it’d have to work differently.)