You can build and target [Any CPU] but there is not guarantee that it will work, and can screw-up deployments. So by publishing both x86 and x64 you support everyone.
x86 will run on 64bit OS but x64 will not run on 32bit OS.
Also if you reference legacy COM inside you managed code you may be forced to build as 32bit/x86 even though your app runs on a 64bit OS, and you could not use my published [Any CPU] version, you would need an x86.
Also if my customer was targeting x64, he would not be able to use my x86 or [Any CPU] published items.