I don't have specific experience with game consoles, but the erratic behavior when powered by the console suggests that the port's voltage is sagging when the Blueretro is attached, possibly lower than what the AMS1117 can tolerate.
A quick search seems to show that the AMS1117 has a minimum dropout voltage of 1v. So for 3.3v output, the input must not drop below 4.3v. Other Low Dropout (LDO) regulators could have a smaller dropout voltage, but that might not be the root-cause.
It's possible that without load, the port provides 4.6-5v. But when loaded, it dips below 4.3v, producing the behavior you see. The problem then becomes: is it the NES that's not providing sufficient current on the voltage bus, or is it the Blueretro trying to draw too much current?
Are you able to measure the port's voltage bus when the Blueretro is attached? That would help prove if the bus is sagging. Does the Blueretro allow you to use USB power when plugged into the console?