Every time we would ask Power Pages to render a non-existing page, we’d receive an error like this:

Instead of the one we were actually supposed to see, which would be this:

So, ultimately, we ended up opening another support ticket, and, yet again (for the second time in a row with Power Pages cases), Microsoft Support engineers were up to the challenge (there was another issue recently which was also resolved by the support quickly: https://www.itaintboring.com/power-platform/your-power-pages-are-misbehaving-out-of-a-sudden-on-monday-morning-purge-the-cache/)
First of all, in order to configure Page Not Found, you need to follow somewhat hidden steps which you will find in this article, just search for “Page Not Found” there:
Site Checker configuration issues | Microsoft Learn
What is not mentioned in that article is that your “Home” site marker has to be pointing to the same page that’s been configured as a “Parent Page” for the “Page Not Found”:

Do that, and HTTP 502 will be gone.
PS. As a side note. Am I happy it’s been resolved and we can now see “page not found” as expected? Of course I am. However, this issue just proves, once again, that Power Pages is a black box, and you are basically at the mercy of the product team when it comes to what works and what does not, yet why and when.
PPS. How did it happen that our Home page site marker was not set up properly? Basically, we are not using a lot of out of the box features. Instead, we have a lot of JS/Liquid customizations, and, as a result, we had not even realized our Home page site marker was pointing to a sub page until the support engineer asked us to fix it to resolve the issue above.