A big factor for the build size has to do with the link.xml (code stripping) file generated. To be on the safe side, the link.xml generated adds code that you may not really use in your project. To minimize the size, you will need to please also modify the generated link.xml code to remove code that you do not need in your project. This link.xml file has to do with Unitys code stripping. For more details please also take a look at this link -> https://docs.unity3d.com/Manual/ManagedCodeStripping.html
Please let me know if that helps and works for you.