So while multiple clients edit the content, we only send changes through websockets and other clients’ editors merge new changes automatically.
But what about saving the content to database? Database service doesn’t know how to merge new changes to what is already saved. When we save to database, do we have only these two options:
send only changes to your server and let the server merge it with what is saved in database and save the result to db store.
send entire document on your editor with request to save in db
None of these options sound good bandwidthwise. I would be really happy to hear what are the best practices around saving to db.
Also should we save as blob or which data type? (please suggest both for mongo and mysql)
(frontent-nextjs, backend-either next serverless, express, or fastify)
There are many ways to approach this, but one of the ways that comes to my mind quickly is to treat the database as something similar to another client that performs an occasional autosave to the database after multiple changes. This is kinda similar to how Figma works, I believe.
The way you store the documents to your databases is up to you, as there are many ways to approach that between serializing to HTML string or to JSON. I would look for a ProseMirror based note taking app on Github to see what they do.
Best of luck! More specifics about the libraries you’re using to do the we sockets would help. Are you using yjs for example?
@colel I will use Yjs with socket.io in express backend for handling collaborative editing and the same user editing a note in multiple devices at the same time. while realtime editing, sending just changes through socket and client knowing how to apply them on their editor is really great.
but if i want to save the content of editor in db as well, this is what i understand that i have to do: fetch latest version of note from db, apply changes in server, update the note with new version. I dont like that i have to send entire note back and forth everytime i make changes. it doesnt seem efficient. thats my problem and I wanna know how to do it more efficiently. changes are made regularly in notes and should be saved with debounce of 1 second max.
and about the data type, shouldnt we save like binary? Idk. Because note sizes can be big and users can have hundreds of notes at least.
lets say I wanna build Evernote clone and I wanna save user notes efficiently. what would be strategy?
(a note: I am not using prosemirror directly, I use tiptap which is built on top of prosemirror. so it shouldnt make difference i believe)
I would be happy to get very technical answer, also names of concepts that I should lookup to understand deeply.
By the way thank you so much for replying it means a lot.
Consider just storing as plain text for now. If you have an issue with the size of the files stored, you can always migrate the data by adding a column or property like "encoding": "gzip", and gzip each text file up into a more space-efficient representation.
As well, it might sound inefficient to do fetching of the entire document, manipulation, and re-storage, but I would push you to try it out and see if there’s an issue with your users. All storage and synchronization should not block the socket connection, so it shouldn’t affect the user’s experience. There are many solutions to making everything more efficient, and those solutions are easier to prioritize when you have many users who are complaining about something. My suspicion is that if you have too many users, that you’re algorithm is causing back-pressure issues or something like that, then you have enough users and a valuable enough product to get a loan or pay for consultation to tackle the very specific performance problem.
I hope that sounds supportive. If you’re just looking for something technically challenging to build, then you might look into orchestration tools like Temporal which can help you scale efficiently and reliably and observability tooling like Honeycomb which can help you identify slow performance areas across services and database calls.
it means a lot.