Landing linear document positions

1

There was this joke on Twitter a while back, which keeps popping into my head as I consider how to announce yet another large breaking change.

USERS: you’re alienating the people who actually use your product
TWITTER: likes are now florps
USERS: what
TWITTER: timeline goes sideways

(The joke being that the Twitter engineers seem very busy making changes, just not changes that anyone is waiting for.)

In this post, I’ll try to convince you that the thing I’m landing on the master branch today is not a sideways timeline, but an actual improvement. What is it?

Positions are now numbers

The gist of the change is that the Pos class is gone, and positions in the document are now described by integers. To interpret such integers, you can mentally flatten the document into a token stream, where a token can be a character, the opening of a node, the closing of a node, or an ‘atomic’ (no content allowed) node. Position 0 points at the very start of the document, and each further position is denoted by the amount of tokens that come before it.

Advantages

  • The main motivation for this change was that it makes position mapping much simpler. Whereas it used to be necessary to perform clever path shifting and re-rooting to map positions relative to a change, you can now think of it as simply replacing some stretches of tokens with new tokens, and pulling or pushing all positions after such a stretch forward or backward a little when the size of the stretch changed. Mapping is now also cheaper (integer arithmetic vs allocating new objects) making having a lot of marked ranges in your document more viable.

  • Certain types of position-manipulating code was really hard to write, even for me, though I invented them. This was mostly seen in the code that implements transformation steps and commands. If you needed to go from the position of a join to the position inside the newly-joined node, for example, you’d have to write some taxing code. Now you can just subtract one from your position.

  • These reductions in complexity have allowed me to finally rewrite the replace transformation step, which I had found myself simply unable to properly do with the old abstractions. This rewrite fixes some issues and paves the way to nice clipboard-related APIs and a more expressive way to specify the allowed structure of documents (#220).

Disadvantages

  • Massive backwards incompatibility. This touches everything that does anything with a position other than simply passing it around. Making the change required me to rewrite about half the codebase (which did give me the opportunity to clean some things up).

  • Position values themselves now confer very little information – they are just integers, and in a non-trivial document it requires a lot of counting to figure out what they point to. Which brings me to…

Using linear positions

The idea behind the old path + offset representation for position is that the representation is close to the meaning, which is often helpful. We lost that, but a meaningful representation is only a method call away. You can do doc.resolve(pos) to get a ResolvedPos object, which bears some resemblance to the old Pos class, but contains more information, in a more easily accessible way – it knows about all the nodes, not just the offsets, on the path from the document root to the position, and has a number of convenience methods to help with common tasks, such as finding positions related to the resolved position.

(Resolved positions are cached, so resolving the same one a lot of times is cheap, and you are encouraged to pass around raw integer positions, not resolved position objects.)

As I mentioned, you’ll need to port any code that treats positions as non-opaque. Good things to grep for are .offset, .path, .shorten.

The document representation stayed largely the same, except that fragments (node content lists) now store their total size (under .size) and nodes expose their ‘skip size’ (the size they take up in their parent node) as .nodeSize. Descending the document tree to find a position is done by searching each parent for the child that contains the position, and then entering that. (This may be optimized for large nodes in the future.)

What else changed

The Node and Fragment APIs shrunk a bit. Node iterators were dropped, in favor of direct indexing. Some convenience methods that were no longer used by the core (inlineNodesBetween, toArray, splice, replaceDeep) were also dropped.

The meaning of Node.slice was changed – this now returns a Slice object, which is the thing you now pass to Transform.replace, and the thing that is used to represent clipboard content. Nodes have a .replace method that is used to replace a range of the node with a slice, and supersedes much of the existing node-updating interface.

I’ve updated the reference guide to reflect the new interface, and will be rewriting (and extending) the other documentation in the coming weeks.

And a warning: Since this change involved rewriting huge chunks of code, it’s likely that new bugs snuck in. The tests are running beautifully again, but it’s likely some code they don’t cover is still broken.

Enjoy. I know incompatible changes are awful, but I’m quite happy to have pulled this one off, and excited to be able to move forward with the stuff that was blocked by it. Reply with any questions or concerns you have.

2

Hmm, inlineNodesBetween was something we used heavily. I guess one should now use nodesBetween and then check each node if it is an inline node?

Being able to look at the length and contents of the path and the the offset is also quite important to us. For example, I need to check whether the path is longer than 0 or 1 and if it has a length of at least 2, I need to do some specific things to the UI based on the value of this 2.

From what I can tell, I cannot get at the path any more, not even used the ResolvedPos[1]. Is that correct? If so, what is the proposed alternative? Do I have to walk up the entire document tree in order to find out what the second element in the path is, or is there some smarter way to do this?

[1] http://prosemirror.net/ref.html#ResolvedPos

3

Reading the new documentation, I think you should be able to get the length of the path using the depth property (this was already possible before when working with Positions) and getting an element higher up in the hierarchy using the node method.

4

Yes, what @kiejo says, plus the index method to get the indices into parent nodes at the various levels. It’s all there in the reference documentation.

5

Yes, I realize it’s in the documentation and that I can walk up the tree like that. The question is if that is the recommended way of doing it. I kind of need to do it at least once for every change.

6

Yes, this (resolving positions) is the way things like that are done in the linear API.

7

I started testing all the basic editing functions in the editor and noticed quite some uncaught errors when writing content. Many of them are probably due to the big changes you described. I’m now trying to break these errors down into reproducable steps and started filing them as issues.
I hope this will help fix all issues related to basic editing. The expected behavior and goal is to have no uncaught errors during normal content editing, right? Or are there currently errors that are specifically thrown on purpose so that a user of the library can handle them in a custom way?

I’m asking this as some of the errors do not seem to have a direct consequence for the user as the expected content is produced even though an error is thrown in the background. Other errors are more critical as they cause unexpected editor behavior for steps that happen after the error.

Here are a few examples of the kind of errors I’m talking about: #280, #279, #266

8

Yes, definitely. API methods might throw when client code calls them in an invalid way, but regular user input shouldn’t.

Thanks for filing bugs! I’m looking through them now.

1 Like
9

I know speaking on the record about stability on a project that is only officially in beta might not be that palatable but it would be nice to get an update on your thoughts about that.

Any other big refactors or re-writes on the horizon? Feelings about how stable you think the overall API will be moving forward?

10

There’ll be more breaking changes, yes. I know that’s a pain to deal with, but I’m still in the process of searching for the proper API concepts, and I don’t think I have quite gotten there yet.

11

How about finding which path index number at any given level?

Do I do:

function findPathIndex(pos, level, pm) {
    const resolvedPos = pm.doc.resolve(pos)
    const upperNode = resolvedPos.node(level)
    const lowerNode = resolvedPos.node(level + 1)
    let index = 0
    while (upperNode.child(index) !== lowerNode) {
        index++
    }
    return index
}

Or is there some more convenient/efficient way to get at that? I assume that the path that is currently available as part of the resolved position is only meant for internal PM use, right?

12

That looks a lot like ResolvedPos.sameDepth.

13

Also, don’t compare nodes by identity to determine whether document positions are the same – the same node object may appear multiple times in a single document.

14

Are you sure?

What I am trying to find is the index number I previously had in the path. I have one editor instance with say 50 different footnotes. Each footnote is a direct child of the doc of that editor. The user has made one change in the footnote editor, and I am trying to find out which of the footnotes the change was made to (for example: footnote 37).

Previously I could simply look at the path, which would be something like [37, X, Y, Z]. The 37 was a clear indication that the change had been made in footnote 37.

Yes, I found out about this earlier. But given that the user just made a manual change to that particular footnote, it should be unique and this should therefore coincidentally work, right?

15

You can still call resolvedPos.index(0) to get the index into the top node that a position points into.

And yes, in some cases comparing nodes by identity is accidentally safe, but you know, might as well get into the safer habit of comparing start positions.

16

ah yes, index() worked. Somehow I missed that. Thanks!