2

I'm doing some research and I need to download a very large number of git repositories, something like 17k+. I wrote a very simple Python script to automate the cloning routine from a dataset containing the GitHub URLs.

first_10 = data.url
name = data.project_name 

for x, i in zip(first_10, name):
    os.system('git clone {} D:\gitres\{}'.format(x,i))

It just iterates over those two pandas dataframe columns for the URL to be cloned and the folder name.

Here comes the problem: every time the script finds a URL that no longer exists on GitHub, the script halts its routine, asks for credentials and won't resume until my input. Doesn't matter if I input correct credentials or gibberish, it will do this every time it finds an invalid GitHub URL. How do I stop git from asking those credentials?

Yun
  • 2,536
  • 6
  • 7
  • 26
Nujabes and Chill
  • 27
  • 3

4 Answers4

1

The reason GitHub sends you a 401 to prompt for credentials if the repository is missing is because they don't want to leak whether a private repository exists. If they didn't prompt, you could easily determine that the repository does exist by getting a 401 and that it doesn't by getting a 404. Instead, GitHub always prompts for credentials, and only then returns a 404 if the repository doesn't exist or isn't accessible to you.

If your desire is not to be prompted at all, as torek mentioned, you can simply set the environment variable GIT_ASKPASS to false and this will work. You could also set GIT_TERMINAL_PROMPT to 0 and that would prevent any prompting for credentials.

However, I strongly recommend that you do indeed set some credentials because GitHub will much more aggressively rate-limit you if you don't set any credentials, and if you do end up using an excessive amount of resources, it's much easier for GitHub to contact you about the problem and ask you to fix it, rather than just block you or make an abuse report to your network provider.

On that note, your Python script is not likely to handle the case where you have a large number of failures for that reason, so you should strongly consider handling that case most robustly. In general, anyone making a large number of HTTP requests to any server needs to learn to gracefully back off.

If you decide you do want to pass credentials, you can do so from the environment using a custom credential helper, or you can use an SSH key and SSH URLs to do this.

bk2204
  • 48,483
  • 5
  • 42
  • 63
0

I'd suggest you've overspecified the problem (turning this into an XY problem): you don't specifically need to make Git not ask for credentials since you could instead merely clone those repositories that do exist.

That said, there are two ways to prevent Git from asking for credentials:

  • Use a URL that cannot take credentials. (Any given server may or may not accept such URLs. With GitHub, you could try to log in as git@github.com via ssh, and present a valid public key. After ssh has authenticated you, Git will give you access to any accessible URL, and deny you access to any inaccessible or invalid URL, without asking for further credentials.)

  • Supply a credential helper that never actually provides any credentials, without asking for any. For instance, you could run with GIT_ASKPASS=false in the environment. See the credentials documentation for details.

(There's one more option as well, which is to allow Git to ask for credentials but redirect the input to a program. This is trickier than just overriding GIT_ASKPASS so there is no reason to cover it here.)

To solve the problem better, find a way to list out the repositories that do exist, and do not attempt to clone the ones that don't. This is likely to go significantly faster.

torek
  • 389,216
  • 48
  • 524
  • 664
0

My guess is that you are using https:// data urls. If you use a personal access token, then GitHub shouldn't be asking you for a username/password. Take a look here on how to set it up.

I think that if you use ssh:// data urls instead, then you wouldn't encounter that problem because git defaults to using your ssh-key for authentication instead of password.

pip1726
  • 148
  • 1
  • 8
0

You probably want to check that the repo exists before attempting the clone. There are answers on Stack Overflow for this here.

Alternatively, if you switch to using subprocess instead of os.system, you can simply "trick" it by reading input from /dev/null which will prevent the prompt - that way, no intervention will be needed and the invalid URLs will simply be skipped.

for x, i in zip(first_10, name):
    subprocess.call(['git', 'clone', x, i], shell=False, stdin=subprocess.DEVNULL, start_new_session=True)
costaparas
  • 4,779
  • 11
  • 15
  • 26
  • Hint: `git ls-remote https://github.com/user/repo` asks password for non-existent repo. – phd Dec 06 '20 at 00:26