I’m looking for general guidelines outlining what a server enabling prosemirror collaboration needs to handle. I am aware of post requesting the website server code but there is definitely a need to document how to generally do it if it is going to be implemented on multiple different stacks.
For my purposes I would like to attempt implementing it within a Meteor.js package. This would enable people to try out collaborative Prose Mirror instances in their own environment without almost no effort. (Given that it is possible to integrate it.)
Have you read the blog post Collaborative Editing in ProseMirror? It goes into more detail about the algorithm used to synchronize changes and how document versions are managed. The information provided there in combination with the code from the website helped me a lot in implementing a socket based version in Node.js.
There will definitely be proper docs about this at some point, but I’m prioritizing development over documentation for now. Given that, @kiejo’s advice is probably the best way to go for now.
Thanks. I figured that proper documentation was something that wouldn’t be coming for some time. Investing a lot of time to create full documentation on a project that is changing certainly doesn’t make sense but maybe investing a little bit of time to ward off these type of questions and help the community would?
Even just a high level bullet list of things that you need to handle/address, pointers to applicable code, and any gotchas to keep in mind I think would be more than enough for now.
Unfortunately my solution is not open source, but I can share some details on the implementation. The overall syncing logic is very similar to git when using rebase:
The server has a master version of a document and clients push their changes, which can only be applied if they are working with the latest version (same version as the server). If they do not have the latest version, they must first get and apply the latest steps from the server and then apply their own steps on top (basically like a rebase). When the server successfully applies new steps, it notifies all other connected clients about the new version, which allows them to request a sequence of steps relative to their own versions (like a pull in git).
For this to work, the server needs to save a history of steps alongside the complete document node.
On the client you will need the following methods from the collab module:
on(‘mustSend’, callback): to trigger the sending of local steps
sendableSteps(): returns a sendable object containing version and steps, send this to the server
confirmSteps(sendable): call this after the server successfully applied the sent steps
receive(steps): call this to update the client to the latest version using steps retrieved from the server (this will automatically apply local changes on top of the received steps)
On the server you will need the following methods:
step.apply(docNode): you will need to iterate over all new steps and apply them to the document node (note that this methods returns a new node and does not mutate the passed docNode)
calculation for the number of steps to send to the client: e.g. server.version - client.version
And the following de/serialization methods:
Step.fromJSON(schema, step)
schema.nodeFromJSON(docNode)
There’s more stuff you will need to do, like throttling the sending of steps and handling the case when there are not enough historical steps on the server (in that case you need to load the whole document). You will also need to handle your state on the client to make sure to wait for the server response before trying to send new steps etc.
But I hope this helps you get started and gives you an overview of how it works. For the details I would still suggest that you look into the website repository as this is where you’ll see exactly how to use the methods.
I didn’t have the time to draw a diagram, but it could be represented pretty nicely as a state machine.
UPDATE: Note that I did not use any of the features related to comments like it is done in the website example.
In the release post for the npm package you allude to the rising importance of people using the API and looking for bugs. I wanted to check in and see if that updates or changes the priority for the subject of this thread?
I was able to get some things working with @kiejo’s advice but I didn’t want to invest a ton of time until there was a bit more stability and you gained a bigger appetite for really outlining what constitutes a good and complete solution for the server-side (with or without the community’s help).
Oh, looks like I forgot to add reference docs for the collab module. I’ll try to rectify that tomorrow. There’s a bunch of guides I want to write (more human-readable descriptions of parts of the system), and wiring up collaboration is definitely one of the things that deserves a guide. Just, these things take time, and I’m not sure I’ll prioritize it over working on tables.
But yes, things are stable now, and once I deliver the collab docs, you should feel safe writing code against that API.
OK cool. I definitely see the value in hammering out the client-side functions including tables before any of the server-side stuff. I’m just trying to keep on top of where this falls in the grand scheme. Once the collab docs start taking shape I’m hoping to start wiring something up in Meteor. If that’s successful it could easily be the quickest way for getting up and running with a full collaborative implementation of ProseMirror.
@funkyeah: I have implemented it with a server that doesn’t understand the JSON, using tornado. The server simply keeps the diffs and received a “full copy” of the document every two minutes, and if at all possible at the moment the user leaves the editor. It keeps both version numbers of the latest full document and the incoming diffs and keeps at least all the diffs since the last full version, if not more. Should the client not be able to send in a diff when the connection is closed, the next user will simply receive the last full version + all the extra diffs since then to apply right at the start of the editing session. (Example: Last diff is diff #214 and the last full document that was received was #200. So the server sends the version #200 + the latest 14 diffs.) I know, @marijn prefers for the server to be DOM aware, but if one cannot do tht, this is an alternative. There may be bugs still, but generally it seems to work: https://github.com/fiduswriter/fiduswriter/blob/3.0/document/ws_views.py (there is also comment management, chats, etc. in there)
this would be awesome, I was thinking along the same lines with using Meter’s pub/sub system to make streaming changes really easy. What are your thoughts on the exact approach? I’m thinking something similar to what johanneswilm mentioned
when a client connects, it fires off a Meteor method ‘getLatestSnapshot’ to ask for the latest snapshot of the document. the server can save these every minute or so
the client gets that document which has a version, say #200 to keep in line with the example above. then the client subscribes to a publication with a query of all transforms for that doc after #200, and applies then sequentially when received by hooking into the collections “oberseveChanges”. now the client has the latest version of the doc
on the client’s prosemirror ‘changed’ event, the client calls a Meteor method “applyTransform” with the transform and the version # that client is applying the transform to, and the client stores that transform as a temp var, something like “transformAttemptingToApply”
if nobody else has applied a transform to that same version #, apply it on the server, which will send out updates to each subscribed client, and then clear out the client’s “trasnformAttemptingToApply” var if it didn’t receive an aerror
Now here’s where it gets a little tricky. If another client had already applied a transform to that version #, throw a Meter.Error with a message letting that client know that they need to get the latest version. At this point, the client needs to revert “transformAttemptingToApply” transform before applying the new transform from the publication. The client then sets a flag saying it’s waiting for the next pub data to come in, and when it does the client then calls the “applyTransform” method again. Rinse and repeat until that change gets in successfully.
It seems that since the “collaborative editing guide” was written, the API has changed.
Is there any new guide, or otherwise some working code which demonstrates its use?
(I’m not sure, but it seems that the website demo is still using the old API)
@amelius I can’t give the definitive answer but I would be very surprised if any real effort was put into updating guides or example code that relates to Prosemirror code presently in the master branch. The linear branch represents a fairly significant change to the codebase (branch is 150 commits ahead, 37 commits behind master) which I would imagine would have some impact on the collab guide and example code.
ejfrancis and a couple others have been putting together a Meteor package that enables collaborative editing: https://github.com/prosemeteor/prosemirror Still needs plenty of work but it might be the most complete open collab lib that works with prosemirror. At any rate we pretty much stopped updating the prosemirror dependency until this change drops and looks to be somewhat stable.
Okay, thanks, good to know. I’m wondering, has the API become significantly more complicated than illustrated in the (outdated) docs? Has there been an overhaul of concepts? Or is the new API only different from the old one in e.g. naming conventions and such? In the latter case, perhaps I can try to figure it out by myself.