Whenever I want to use an inline node view it will be re-rendered on enter. The red box is a node view.
I think this is because it’ll be wrapped in a new paragraph so it’s not possible for ProseMirror to keep its original element? Can I do anything to change that behavior?
Your diagnosis of the issue is correct—ProseMirror doesn’t move nodes around in its view tree, so if something gets a new parent it is re-rendered.
You could, if you really want to, add your node view to a cache on .destroy(), and then when a new node view is created for a matching node, take it from that cache instead of re-creating it.
Thanks for that idea @marijn! I tried implementing such a caching system. Basically this is working fine but there is one problem:
When I press enter the followed node view in this paragraph will be destroyed and re-created in the new paragraph as expected. But when I press backspace the new node view will be re-created before the old one is destroyed, so I’m not able to cache it. I hope there’s a good reason for that? 
Well, that’s a symptom of the update algorithm working front-to-back, first filling in the grown node before it gets to the now-deleted parent node and calls destroy on its children. Nothing deep, but hard to change without fundamentally rethinking the (already complex) update code.