Tracked Changes with Strict Document Format

1

@marijn in an rfc pull request (https://github.com/ProseMirror/rfcs/pull/3) you suggested tracked changes be done by storing steps like in https://github.com/prosemirror/prosemirror-changeset. I actually did take a look at that and wanted to go in that direction originally but due to my requirements it seemed like the wrong approach.

I am working with an external system that already creates documents with tracked changes. I need to support that document format and be able to output the same format. This existing format wraps inline text in elements and places attributes on nodes. And then inserts corresponding metadata nodes into the documents <head> element. There are more than just insert and delete operations as well. Wrapping/unwrapping content, attribute changes, and replaces are all forms of tracked changes. I was uncertain how to or if it was even possible to use prosemirror-changeset with the above constraints.

So I took the approach of using marks for the inline text changes. I am still working out a method for applying attributes to nodes that have a tracked change. I am applying these marks by hooking into input handlers such as handlePaste, handleDrop, handleTextInput, and handleDOMEvents.cut. I am worried that I wonā€™t catch every input type. Iā€™d like to be making changes at a lower level like in prosemirror-changeset.

Does my approach lead to madness? Do you think there is still a way to leverage prosemirror-changeset? FYI these documents are also edited in a collaborative environment.

2

Probably, yes, but Iā€™ve never worked with a system like what you describe (a document format that encodes various types of changes), so trying to do it differently might still lead to madness.

But in general, I think you really want to work on the step or transaction level, not the UI level, since as you already hint, thereā€™s just too many different ways in which people might interact with your documents, and thereā€™s a non-trivial amount of interpretation going on in methods like Transform.replace when you do things like paste or delete arbitrary selectionsā€”it might have to introduce nodes or unwrap nodes to conform to document schema constraints. So if you want relatively regular, reliable data, the transaction level is what you want.

Whether prosemirror-changeset is a good fit here I donā€™t knowā€”it reduces a set of steps to a series of insertions and deletions, but if you have other types of changes that you want to track, it may not be easy to apply.

3

In my editor I am showing tracked deletions as text with a line struck through it. And insertions as a different color than the rest of the document.

Because this is a collaborative document I need to display those changes the same in all peer editors. To do this I believe the tracked changes need to be applied directly to the document in order to be sent to the peer editors. That being the case I think appendTransaction could be the best place for me to apply the tracked change marks and node attributes. But I am not entirely sure how best to go about it. I think I would have to go through each transactions steps to figure out if an insertion or deletion happened and then apply the marks to the ranges of those steps. Now correct me if I am wrong but I think I could use prosemirror-changeset here to calculate the inserted and deleted spans I need. Using those spans I could apply the marks/attributes to the spanā€™s ranges.

Does that sound feasible?

4

The ā€œchangesā€ are implicit in the steps youā€™re already sending to peersā€”i.e. if you derive the deleted sections from the set of steps since a given point in time, all peers should be seeing the same steps and compute the same deletions.

5

Iā€™ve been moving forward with this method and it has been working well so far. I came upon something that I do not quite understand though. When lifting an empty block the changesetā€™s insertion position range appears to be off by 1.

In my example I have a list with 3 items. I hit enter to create a 4th item. Then hit enter again. This causes the lifting of an empty block and ultimately a ReplaceAroundStep. The resulting doc is a list with 3 items and a paragraph after the list.

Before:

<ol>
  <li><p>Item 1</p></li>
  <li><p>Item 2</p></li>
  <li><p>Item 3</p></li>
  <li><p></p></li>
</ol>

After:

<ol>
  <li><p>Item 1</p></li>
  <li><p>Item 2</p></li>
  <li><p>Item 3</p></li>
</ol>
<p></p>

The new document is correct but the span in changeSet.inserted points to the <ol>. So if I were to run newState.doc.slice(span.from, span.to) it would give me the <ol> instead of the <p>.

I am grabbing the change set in Plugin.appendTransaction like this:

var changeSet = pm.changeset.ChangeSet.create(oldState.doc).addSteps(newState.doc, transaction.mapping.maps);

Iā€™m not sure if my assumptions of change set are incorrect here or if the insertion spanā€™s range is incorrect. Other transactions have worked with my assumptions up to this point.

6

Iā€™m not entirely sure what your assumptions are, but what lift will do, in this case, is create a replace-around step that replaces the opening list item token (<li>) after the empty paragraph with a closing list token, and deletes the two closing tokens after the empty paragraph (</li></ol>). That way, the paragraph itself isnā€™t changed or moved, but only the tokens around it are updated to reflect the new structure.

7

My assumptions were that the change setā€™s inserted spans would point to the new <p></p> after the <ol> in the newState parameter of Plugin.appendTransaction. And that the change setā€™s deleted spans would point to the old <li><p></p></li> in the oldState parameter of Plugin.appendTransaction.

However I am getting these results (FYI in my example there is more content before the list making the positions start in the 500s):

var changeSet = pm.changeset.ChangeSet.create(oldState.doc).addSteps(newState.doc, transaction.mapping.maps);

changeSet.inserted[0]
// { from: 545, to: 546 }
newState.doc.slice(changeSet.inserted[0].from, changeSet.inserted[0].to)
// <ol></ol>

// changeSet.deleted has two items in it

changeSet.deleted[0]
// { from: 545, to: 546, pos: 545 }
changeSet.deleted[0].slice
// <ol></ol>
oldState.doc.slice(changeSet.deleted[0].from, changeSet.deleted[0].to) 
// <ol></ol>

changeSet.deleted[1]
// { from: 548, to: 550, pos: 548 }
changeSet.deleted[1].slice
// <ol><li></li></ol>
oldState.doc.slice(changeSet.deleted[1].from, changeSet.deleted[1].to)
// <ol><li></li></ol>
8

Iā€™ve run into another unexpected change set involving lists. It happens when deleting an empty list item with backspace.

Before:

<ol>
  <li><p>Item 1</p></li>
  <li><p>Item 2</p></li>
  <li><p>Item 3</p></li>
  <li><p></p></li>
</ol>

After:

<ol>
  <li><p>Item 1</p></li>
  <li><p>Item 2</p></li>
  <li>
    <p>Item 3</p>
    <p></p>
  </li>
</ol>

The changeset contains one deleted span and no inserted spans:

// DeletedSpan
{
    data : undefined,
    from: 1499,
    pos: 1499,
    slice: {content: Fragment, openStart: 1, openEnd: 1},
    to: 1501
}

where the slice is two list_item nodes with no content.

I would expect to get a change set containing one deleted list_item and one inserted paragraph.

Looking more closely at prosemirror-changeset leads me to believe it is unable to understand the nuance of a backwards join (which I believe is happening in this case). The transaction here is one replace step

// ReplaceStep 
{
    from: 1499,
    slice: {content: Fragment, openStart: 0, openEnd: 0}, // An empty slice
    structure: true,
    to: 1501
}

And transaction.mapping.maps (which I pass into ChangeSet.addSteps) has only one StepMap

// StepMap
{
    inverted: false,
    ranges: [1499, 2, 0]
}

I do not think that ChangeSet can do anything with that one StepMap other than find one deleted span.

@marijn is prosemirror-changeset returning the correct results? It certainly seems to be returning the only results it can given the input. But it does not really seem like the correct results to me. If prosemirror-changeset is working correctly then it looks like I will need to approach the detection of changes differently and not solely depend on prosemirror-changeset.

9

It deletes the </li><li> tokens, which can be done with a single replace step.

It doesnā€™t do nuancesā€”it represents everything as only deletions and insertions, which is correct (it fully expresses what happened), but may not always be easy to interpret/display.

10

Thanks for bearing with me @marijn. I think I understand what prosemirror-changeset really does now. And Iā€™ve written some code that seems to do a good job at detecting a backwards join using the change set data and some looks up into the old and new editor state.

11

Hey, Iā€™ve taken a good look at prosemirror-changeset. This looks like an interesting and fresh approach at tracking changes. But if I have understood it right, given that it relies on diffing to one particular document in the past, there are a few inherent limitations. Most notably, it cannot track more than one change per text element - it either tracks that a specific piece of text has been added or that it has deleted. It cannot track that author 1 first wrote ā€œhelloā€ and that later author 2 deleted that text, right? And for collaboration, where clients connect/disconnect at different times one will need to do some manual work to ensure that everyone has all the steps and starts out from the same document.

I was thinking of another approach, more similar to how common wordprocessors do it: Add a mark to all newly added inline content, and when the user tries to delete content, prohibit the deletion and instead mark it with a deletion mark. Additionally, add marks to paragraphs that the user tried to merge with preceding paragraphs, and add ā€œchange styleā€ marks to content that has change style. But I wonder how one would should do that in ProseMirror in a sane way. Especially the deletions seem to be an issue.

One could try to use filterTransaction to stop the transaction from taking place. But then one somehow needs to trigger an alternative transaction at the same time without letting ProseMirror get confused. Starting a transaction from within a transaction is probably not a good idea.

Another option seems to be to use appendTransaction to try to undo the deletion and then instead do the marking. But also that does not sound so good as deleting content and then re-adding it seems a bit unnecessary.

@marijn do you have any suggestions for what would be a good way to essentially replace transaction steps with alternative steps within a plugin?

1 Like
12

What I ended up doing was the following (as part of a plugin):

filterTransaction(tr, state) {
  let newTr = state.tr
  ...
  setTimeout(() => {view.dispatch(newTr)},0)
  return false
}

This seems to be working. It stops the current transaction and quite immediately thereafter it dispatches a new transaction instead. Just make sure to check in the filterTransaction function that you donā€™t stop the newTr in the next round.

1 Like
13

Youā€™ve made dispatch asynchronous, which might lead to ā€˜mismatched transactionā€™ errors when the user, for example, types very fast.

14

Itā€™s asynchronous with a timeout of zero, the point of which is so that everything else connected with the last transaction is called first, and then the new transaction is dispatched immediately thereafter. I have not been able to create mismatched transaction errors, but that would of course not be good. Do you have a better solution?

I guess one could instead always invert all previous steps of the transaction first and then reapply them in a changed form. I thought that sounded like a lot of unnecessary adding of steps and was concerned about performance. But now that I saw that this is basically also what happens with unconfirmed steps, Iā€™m less worried.

15

I guess the asynchronous part was not a problem as I was denying all transactions unless they had a meta set that they had been generated by the fitler. Given that JavaScript is single thread, that meant that they would still come in in the right order once they were processed.

However, I agree this was not that beautiful and I changed to to where it is first reverted the steps and then reapplying them in a new transaction instead. Itā€™s a lot more steps than it used to be, but it appears to not have any significant influence on speed.

1 Like
16

I actually managed to hack this a little bit on a small thing I was working on. I wanted to stop cursor selection, and filtering was the easiest way to do it rather than trying to reset the selection, but I wanted to update some plugin state here. If I filtered the transaction then the plugin would never know the user had tried to cursor, so the following seemed to work Object.assign(tr, new Transform(state)) inside filter transaction, which basically mutates the tr by assigning all its properties to their defaults, then you can go from there (I just called setMeta after this). Itā€™s not the nicest fix but I think itā€™s less error prone than setTimeout, although somewhat prone to breaking if the implementation changes.

17

Iā€™ve created an RFC for replaceTransaction that aims to solve your use-case (among others), so Iā€™d be keen to get your feedback. https://github.com/ProseMirror/rfcs/pull/10

/cc @rich