Just watched a YouTube video on this, your thoughts are correct. There are more ancestors than people ever alive. 100%. We'd need to find humans on another planet if we want new blood, that evolved independently from different shoots.
If you were to subscribe to the concept that it all started with the equivalent of Adam and Eve (even if you don't subscribe to the religious aspects), then it would indeed seem 100% likely that we are all related.