The first step would be to manually identify some samples that would be useful, and some that are useless. You may have to do this in two sub-batches, the first being to more sharply define what you you are trying to exclude (and why). Some possible reasons to exclude are that there are random car horns overlapping human utterances, or there is constant jackhammering throughout some recordings.
Ultimately you would be looking for settings that give you the best results for textgrid-to-silences, which most closely identifies "speech" versus "not speech". In the jackhammer condition, everything is "speech", there is no silence. With a set of usable threshold values for silence-detection, you can (with a script) look at the pitch and amplitude listings to find sufficient amplitude distinctions over reasonable time periods (e.g 15 dB drop for at least 1 second) also correlated with pitch difference (looking for "no pitch"). This may characterize the distinction that you're interested in – "not worth spending time on" vs. "can get something from this".
Gender is such a complex topic that I don't see any prospects for blindly assigning speakers to a gender.