Seeing my neighbors gathering ginkgo nuts made me curious enough to try them, and I waded right in without understanding the risks! TLDR— they're not a great food source. It's yet another one of those cases where you have to wonder what "delicacy" means.The actual fruit (looks like a rotten plum, smells terrible) has ginkgolic acids which cause contact dermatitis (think poison ivy).
Then the nuts themselves contain Ginkgotoxin, which interferes with your B6, screwing up your nervous system and causing seizures. Cooking reduces but does not eliminate Ginkgotoxin.
I only ate one, and ate it raw. It was a delightful texture, but tasted like chewing random plant matter. Like leaves from a tree. Was maybe half a cubic centimeter of matter. Escaped any ill effects.
According to my research, kids can have seizures from as few as 10 nuts, which would probably be like 1.5 spoonfuls if you mashed them up. The guidelines I found don't seem very scientific but supposedly a kid can safely handle 3-5 nuts over the course of a day, and an adult could handle 5-10. So it doesn't seem like there is a good margin of safety.
Overall a real risk to health for an insignificant amount of food that doesn't taste special. But a nice texture.