Request: Make assets easier to diff in source code form

[kalms]

Hi,

We are storing all our assets in text format. I have some suggestions which will make it easier to diff assets using text-based tools:

Store the serialized flow graph as pretty-printed JSON instead of a single-line string
For people who use a proper VCS (git or somesuch), and who have configured Unity to store assets as text, this gives those people a reasonable chance to understand the impact of their changes from a text diff in the VCS tool; is the change just re-wiring a few connections? is the change adding and removing nodes? something else major? Also, perhaps unrelated changes from two different sources can be merged as text? This does rely on FlowCanvas’s serialization being stable though, which is what the following item is about…

Have a persistent ID for each node
I haven’t looked into this deeply, but it appears to me as if FlowCanvas will re-do ID assignment during deserialization (in GraphSerializationData.Reconstruct()) and whenever changes are being made to the set of nodes or the connections between them (in Graph.UpdateNodeIDs()). This means that if a node with an ID ‘in the middle’ is removed, all subsequent nodes will be re-numbered. This results in a lot of changes to the serialized representation even though there are little or no functional changes to the flow graph. If there is a persistent ID for each node, and the set of nodes are always serialized/deserialized in node ID order, then nodes will remain in the same order in the JSON representation, thereby reducing the amount of unnecessary change in the JSON compared to the main data. (If it is important for Node.ID to get re-numbered, it would work perfectly well to have an extra serialization-ID that persists and is only used for ordering things at serialization/deserialization time.)

Example
The attached archive (Sorry, cannot upload: forum disallows both JSON and ZIP attachments) contains two JSON files which show the before-and-after differences when I opened & saved a flow graph which another colleague had created, and someone else subsequently has made a tiny change to one custom node.

The following block is moved in the file even though there is no big functional difference to it, probably due to nodes being renumbered:

{
“_inputPortValues”:{

},
“_nodeColor”:{
“r”:0.7490196,
“g”:0.4980392,
“b”:1.0,
“a”:1.0
},
“_position”:{
“x”:5791.0,
“y”:5705.0
},
“$type”:”FlowCanvas.Nodes.DoOnce”,
“$id”:”15″
},
{
“_inputPortValues”:{
“Wait”:{
“$content”:false,
“$type”:”System.Boolean”
},
“Rotate Towards Target”:{
“$content”:false,
“$type”:”System.Boolean”
}
},
“_nodeColor”:{
“r”:0.7490196,
“g”:0.4980392,
“b”:1.0,
“a”:1.0
},
“_position”:{
“x”:5926.0,
“y”:5593.0
},
“$type”:”PlayAttackNode”,
“$id”:”17″
},

There are a number of $ids that get renamed, even though there is no functional difference to the graph.

This block, however, is a legit change, caused by changing the definition of one custom node:

“Speed”:{
“$content”:2,
“$type”:”System.Int32″
},

=>

“Speed”:{
“$content”:0.0,
“$type”:”System.Single”
},

With a stable serialization implementation, the only diff between the files would have been the last block.

[kalms]

I looked into this a little bit myself. It was quick to change all places which call Graph.Serialize() to always ask for JSON pretty-printed formatting at serialization time. That didn’t help a lot though; Unity encodes the pretty-printed multiline string as a long single-line “line1\r\nline2\r\nline3\r\n…” type string, and I see no way to convince Unity to split a string across multiple lines in the .asset file. Without the newlines in the .asset file, the pretty-printed JSON is not very helpful for diffing.

[Gavalakis]

Hello,

Thanks for the follow up. I’ve also tried this myself for the shake of testing your suggestion out, and found the same issue you did.
I will try a few things up and see if they work though.
As far as consistent IDs, once after the new version is out, I will indeed take a look at the possibility of having more consistency in the IDs!

Thanks!

Join us on Discord: https://discord.gg/97q2Rjh

[kalms]

Hi,

Some more thoughts on this. This is all based on an older version than 2.70 by the way, so some of these points might be obsolete by now.

1. If you separate out the viewing parameters, i.e. current window and zoom factor, and store those in native Unity serialized fields, then it will be a lot easier for users to tell the difference between “I opened a graph and looked around in it” and “I opened a graph and made some modifications to its content” when diffing .asset files.

2a. I understand that this is probably a major undertaking, but, serializing the entire graph with Unity’s Serializable types will instantly lead to the data becoming more easily diffable and mergable. Perhaps there is a way to store the graph description as a sort of a data description instead of a functional object structure, but in Serializable types? It might lead to a need for code generation (generating a Serializable type for each node type, to enable Unity to be able to serialize/deserialize them), not sure.

(2b. Another option could be to store the JSON content in a text file separate from the .asset. It would allow the JSON to be stored as a pretty-printed normal text file. This makes it more cumbersome for users since each graph will be a pair of files instead of just one. Probably not worth it for most people.)

[Gavalakis]

Hello again,

Thanks for the follow up and for the suggestions.

To be honest, out of your three suggestions, you last one (pairing a text asset) is the one I have most prominently though about doing 🙂
It indeed could be a bit cumbersome, but I have some ideas in making it as transparent as possible.

Your 1st suggestion could also be another solution, but as far as the 2nd, this would require a total rewrite and would break old serialized projects, thus unfortunately it’s not something that I can consider doing :).

So probably something between #1 and #3 can be a good solution.
Thanks!

Join us on Discord: https://discord.gg/97q2Rjh

[kalms]

Hi,

I was thinking a bit more about option #2 (native Unity serialization) recently.

I think you could accomplish this by creating an alternative to the FullSerializer, which would be a NativeUnityObjectSerializer. It would contain a set of serializable Unity objects which are just enough to represent the constructs that can exist within a JSON structure. Then, the places in the code where you call JSONSerializer.Serialize() / JSONSerializer.Deserialize(), you would not assign the results to a string field – you would assign it to a serialized-root-object-ref instead.

In order to not break backwards compatibility you would need to keep both the existing FullSerializer and the new NativeUnityObjectSerializer alive in parallel. At load time you would then have to try loading both kinds of data from the asset; at write time you only write one of the types.

It is a fair bit more work than the other options, absolutely. However, I think it is worth considering again.

[Gavalakis]

Hello again,

The problem though is the Unity Json serializer does not have nearly the capabilities of FullSerializer, thus is unable to actually serialize the same data that FullSerializer does and which are required.
Furthermore, even with Unity’s Json serializer, the string will still look the same as it does with FullSerializer (a big non-formatted string), unless I misunderstood something in your explanation 🙂

Join us on Discord: https://discord.gg/97q2Rjh

[kalms]

I think you misunderstand me.
I am aware that pushing the internal complex data structures through Unity’s serializers will not work.

However, if you look at what JSONSerializer does, then via TrySerialize() it walks through complex internal data structures, and translates it to something simpler – a set of fsData objects, I think – and then via fsJsonPrinter.CompressedJson() it converts the fsData tree into a single text string.

For that second step, instead of going fsData -> string, you could go fsData -> an equivalent set of C# classes which are serializable, and then leave a ref to the root of that C# object tree within the class you want to process. You would do all that as part of OnBeforeSerialize(). Then Unity will handle the actual serialization. The results will be well-structured YAML within the .asset file.
(I am not sure but I suspect it would be possible to even make the fsData class native-serializable…?)

Does that make more sense?

[Gavalakis]

Hello again,

Thanks for the follow up and the clarifications to your idea.
I will certainly need to investigate this route further and see if and what can be done. It is certainly not an easy task (not to mention risky) to undertake, as well as promise anything right now, but I will definitely take a closer look at your idea.

Thanks!

Join us on Discord: https://discord.gg/97q2Rjh

[Author]

Posts