Rescaling with swscale: SWS_FAST_BILINEAR or SWS_BILINEAR?

A coworker of mine who is developing the the Apprentice Video Player ran into a weird issue with the quality of resampled/rescaled frames. He worked around it, but I just ran into what I think is the same issue, seeing as my quality was suffering the exact same artifacts his was. I couldn't for the life of me figure out what the issue was, until I looked closely at my selected scaling algorithm: SWS_FAST_BILINEAR.

The documentation between SWS_FAST_BILINEAR vs SWS_BILINEAR isn't very clear. I chose the FAST version because, well, speed is good, right? Not if it ruins your output. I figured the FAST version might be similar to some nearest-neighbor algorithm, while the non-FAST version would be proper bilinear filtering. I opted for the FAST version because I'm not rescaling my video: I want the same frame size, just a different pixel format. I don't really need proper bilinear filtering. Or so I thought.

The difference between the two? Check out the images below. The FAST version is simply unacceptable for any kind of real use. It might be a nifty option if you don't care about quality, and I seriously mean don't care. If you care-at all-, don't use the FAST option.

The original YUVJ444P JPG image

The output image using the SWS_FAST_BILINEAR option to convert the color space to RGB24 (look at the edges of the jellyfish)

The exact same code produced this output image, but with the SWS_BILINEAR option set to convert the color space to RGB24

In general, just don't use SWS_FAST_BILINEAR unless you're willing to accept artifacts like the above ones. And if you're getting weird artifacts, try using a different resampling method.