How to strengthen the code cave infection of aarch64 elf shared libraries?

I am coding a packer for my aarch64 elf shared library.
I embedded a loader for decrypting the text section in a code cave within the PT_LOAD segments.
The rela.dyn entry is modified such that it points to the address of the loader; it this way,
the loader will be triggered once the shared library is called.

The cons of this mechanism are:

  1. The attribute of the PT_LOAD segment is RWE. This raise suspicion since a normal PT_LOAD is not writable.
  2. The address of the loader is written in the entry of the rela.dyn. One can trace the address back to the assembly of the loader and eventually hack my decrypting algorithm. They are plenty of techniques to fool the flow-orientated disassembler for x86. However, the addresses of aarch64 are aligned, I hardly find any relevant techniques.

Are there any ways to strengthen the cons ?

Reference

  1. https://github.com/0x00pf/0x00sec_code.git (elfun/elf_infect.c, code cave infection)
  2. https://github.com/0xN3utr0n/Noteme.git ( triggering loader with rela.dyn )