Ignore robots.txt when using wget
In the JRA project, @mscheinert had to tell wget
to ignore the robots.txt
. This should be a safe default for all scripts.
In the JRA project, @mscheinert had to tell wget
to ignore the robots.txt
. This should be a safe default for all scripts.