The first I think 1mln threads are by now archived fully. b/, /r9k/ and /lgbt/ I think are about 75-90% done Thus far the following boards are fully scraped: Sometimes an AWS instance would get right away 429, sometimes after an hour, sometimes never. I think if you are US hosted you not usually rate limitted. black.host and hostinger seem also unaffected. I think Hetzner and some other company that I forgot also was directly 429.ĪWS/Google seem to be doing fine. I guess they are globally rate-limited as well. I ran a doplet from digital ocean, right away a 429 too many requests. I spin up an OVH Cloud instance - same result. Right now I think the entire OVH network is banned by them, I tried spinning all of my 3 VPS' return 429 or even 500. Yeah,no, imgur, does not support IPv6, thus can't connect with an IPv6 IP to them, otherwise it would have been already over May I remind everyone I accept and support donations?ĭonate via Bitcoin (BTC), our wallet address is 15zTEJnSFet5tSwu6FZefnpqX3hHZw1Gax
Sooo I'm guessing it's well over 1.4k USD by now. + ProtonVPN and NordVPN (so eh 20usd each?) so I can scrape at home with diff IPs whenever the IP gets blocked/rate-limited by imgur (my home IP is i assume perma banned now by them) + about 200USD in ~6diff VPS that I also use to pull data + ~380USD in AWS C2 VPS (0.09USD per GB outbound traffic?!) I switched now to google cloud, but they kinda suspend me every few hours thinking it's a DDoS attack, however they have a 300USD free credit for each new registration, so it's been very helpful overall +220USD for ~50 IPv4 IPs (which all are now blocked by imgur, yay!) It costed me thus far about 450USD for a dedicated server Only AWS is great bandwidth/cpu wise but God it's so expensive.
most VPS/dedi services claim they offer 100mbps but yeh nah. I don't remember when I slept like a normal person. /vint/ - International Vidya ĭunno, so far i've scraped 20TB in about 30days.
0 kommentar(er)
