Wheat has well over a hundred components. These cause a negative cell reaction in most people, also known as auto-immune response. Taking out the gluten leaves the rest of these harmful components in the wheat. Other grains have similar parts that are harmful, but not nearly as many as contained in wheat.
Additionally, "IT" is not just about wheat. Dr. Davis and Dr. Perlmutter and Dr. Amen, among others discuss the harmful results of eating too many carbs. Its about the carbs being recognized as sugar by your cells. The entire issue of your body processing these carbs (glycation) causes all types of nasty effects and you become fat/sickly/foggy brained. These effects start at the cell level and keep stressing your body organs until an auto-immune reaction becomes a serious illness. Your behavior and life style are seriously changed as a result of your health deteriorating.
Gluten is toxic to most people. Adding it to your nut flour baked goods doesn't make much sense unless you like to ingest poison.