Yes this is true, there was a limited population for genetic diversity at some times and places. Also inbreeding is important for certain distinctive features to "cement" into the genome of a group. Look at the modern Amish who are so isolated that they produce offspring which have rare genetic disorders such as maple syrup urine disease. Wars and trade introduce new genetics to groups. The 50/500 rule to ensure genetic diversity is quite apt and in the past many smaller communities failed over time due to becoming less evolutionarily fit. Look at the Appalachians or the clan in Turkey whose children aren't even able to walk.
However, genetic evidence suggests that the current Homo sapiens sapiens species has various hominid species mixed in. Africa is one of the most genetically diverse places in the world and it is most likely that many of the ethnic groups are products of interbreeding with other hominids. There is evidence that some Caucasians are mixed with Neanderthals and some East Asians are mixed with Denisovans. Considering that there is new archaeological evidence of humanoid settlements nearly as old as 400,000 years ago our current historical records and what we know of prehistory is so lacking we cannot yet begin to wonder at the diversity of the human genome and its various influences.