Quoting myself from https://www.mattcutts.com/blog/photo-mosaic-effect-with-go/#comment-102923

You can’t set the pixels of an image.YCbCr (i.e. a JPEG image) directly, because the underlying data doesn’t map 1:1 to pixels, due to chroma sub-sampling. If you know what you’re doing, and are aware that setting one Chroma sample can affect multiple pixels, then the image.YCbCr struct exposes enough to let you set pixels. But if you don’t know what you’re doing, then we figured it’d be less dangerous to disallow behavior (setting pixels) than have potentially surprising behavior.

http://en.wikipedia.org/wiki/Chroma_subsampling

In general, work in RGBA as long as you can. The jpeg Encoder converts to YCbCr just before writing bits on the wire, but the intention is that you draw on your RGBA and pass your RGBA to jpeg.Encode.

By "disallow behavior", I should have said "not easily expose behavior".