Hi,
It happens with simple compositing. Just change the blend mode and got alpha channel artifacts. It haappens with masks too and all layers got affected by this problem, not just the layer with different transfer mode. Tested in 5.11 and 5.14 builds.
Here a screenshot:
https://drive.google.com/file/d/1BbbJ2JEryz86h4pR5SeNox4byvzW-ZOU/view?usp=sharing
And the test file with the images:
https://drive.google.com/file/d/14uVVmFOFORE6Dk0fWwfQmhEzfZ64oXKi/view?usp=sharing