Timeout vs no response from server, how can I separate these?
This question is regarding a bot of mine which’s primary focus is scraping.
The path is mapped out correctly and it does what it needs to do. Rate limits are tested and I am certain this is not a factor, if it was and where it was we received actual responses.
However, the webpage(s) I am trying to scrape seem to have build in a kind of weird/ unfamiliar security manner, something that I haven’t came across before. And here I am wondering, how it’s executed and how I deal with it appropriately.
While the scraper/bot is doing it’s thing, sending requests getting responses, at random times it will encounter this what I suspect is a security measure. There are simply no responses back from the server, not a 4xx error or any at all.
At first sight the proxies just appear dead, but that’s not it, because they are not. The proxies work just fine, and manually I can just browse the page on them, no issues here.
The server just stops giving responses.
Now to find a workaround for this, I would need to be able to tell the difference between a timeout (for my proxies) and a no response. They appear the same, but are not.
Does anyone have insight on this problem, maybe there is a genius way to separate those that I am not aware of.
2 Answers 2
Trending sort
Trending sort is based off of the default sorting method — by highest score — but it boosts votes that have happened recently, helping to surface more up-to-date answers.
It falls back to sorting by highest score if no posts are trending.
Switch to Trending sort
Now to find a workaround for this, I would need to be able to tell the difference between a timeout (for my proxies) and a no response. They appear the same, but are not.
A timeout is if the server does not respond within a specific time. No response means, that the server either closes the connection either before the timeout occurs or that it will close the connection after the timeout occurred without sending anything back.
The first case can be easily detected by the connection close before timeout. If you want to detect instead if the server will close the connection without response only after your current timeout then your only option is to extend the timeout. There is nothing in the server which will indicate that the server will close the connection without response at some future time.
And since your only connection is with the proxy there is no real way to detect if the problem is at the proxy or the server. Your only hope might be to set your timeout waiting for the proxy larger then the timeout the proxy has waiting for the server. This way you’ll maybe get a response from the proxy indicating that the connection to the server timed out.
No response from server что делать
RaceRoom Racing Experience
12 дек. 2017 в 11:45
15 дек. 2017 в 13:07
16 дек. 2017 в 7:15
16 дек. 2017 в 9:20
16 дек. 2017 в 11:23
16 дек. 2017 в 14:00
14 ноя. 2018 в 14:19
This is a workaround for everybody running a dedicated server and albeit other people can join, the dedi host cannot. Usually this is caused by newer routers that don’t support NAT loopback, a necessary function for your network to re-direct http-calls from inside the network.
Hier ein kleiner Trick für alle die einen Dedicated-Server hosten, aber selbst dem Server nicht beitreten können. Der häufigste Grund dafür sind neue Router die NAT Loopback nicht unterstützen, was aber benötigt wird um http-Anfragen in demselben Netzwerk richtig zuweisen zu können. (Deutsche Anleitung siehe unten)
1. Step
⦁ Press windows-key+R to open the run-dialog, enter “hdwwiz” without the quotation marks and hit Enter.
⦁ In the following window click next, on the next page select the second option (should say sth like “select hardware from a list manually”) and click next again.
⦁ From the list select “network adapter” and click next.
⦁ On the next page select “Microsoft” as Manufacturer (left) and on the right side select “Microsoft Loopbackadapter”. Click next and wait until installation is finished.
No response from server что делать
This isn’t really a problem, per se , but it can be awfully confusing. If you use nslookup to look up a type of data for a domain name, and the domain name exists, but no data of the type you’re looking for exists, you’ll get an error like this:
% nslookup Default Server: terminator.movie.edu Address: 0.0.0.0 > movie.edu. *** No address (A) records available for movie.edu.
So what types of records do exist? Just set type=any to find out:
> set type=any > movie.edu. Server: terminator.movie.edu Address: 0.0.0.0 movie.edu origin = terminator.movie.edu mail addr = al.robocop.movie.edu serial = 42 refresh = 10800 (3H) retry = 3600 (1H) expire = 604800 (7D) minimum ttl = 86400 (1D) movie.edu nameserver = terminator.movie.edu movie.edu nameserver = wormhole.movie.edu movie.edu nameserver = zardoz.movie.edu movie.edu preference = 10, mail exchanger = postmanrings2x.movie.edu postmanrings2x.movie.edu internet address = 192.249.249.66
% nslookup Default Server: terminator.movie.edu Address: 0.0.0.0 > terminator Server: terminator.movie.edu Address: 0.0.0.0 *** terminator.movie.edu can’t find terminator: No response from server
% nslookup Default Server: terminator.movie.edu Address: 0.0.0.0 > ls foo. — Try to list a nonexistent domain *** Can’t list domain foo.: No response from server
In this case, no name server is running. If the host couldn’t be reached, the error would be «timed out.» If a name server is running, you’ll see the following error message:
% nslookup Default Server: terminator.movie.edu Address: 0.0.0.0 > ls foo. [terminator.movie.edu] *** Can’t list domain foo.: No information
That is, unless there’s a top-level foo domain in your world.
% nslookup *** Can’t find server name for address 192.249.249.3: Non-existent host/domain *** Default servers are not available
The «default servers are not available» message in the example is misleading. After all, there is a name server there to say the address doesn’t exist. More often, you’ll see the error «no response from server» if the name server isn’t running on the host or the host can’t be reached. Only then does the «default servers are not available» message makes sense.
% nslookup *** Can’t find server name for address 192.249.249.3: Query refused *** Default servers are not available %
This one has two possible causes. Either your name server does not support inverse queries (older nslookup s only), or zone security is stopping the lookup.
In BIND 4, the statement looks like this:
In BIND 8, the statement looks like this:
This statement causes your name server to respond to the inverse query with a «fake» response that is good enough for nslookup to continue.
Zone security is not limited to causing nslookup to fail to start up. It can also cause lookups and zone transfers to fail in the middle of a session when you point nslookup to a remote name server. This is what you will see:
% nslookup Default Server: hp.com Address: 15.255.152.4 > server terminator.movie.edu Default Server: terminator.movie.edu Address: 192.249.249.3 > carrie.movie.edu. Server: terminator.movie.edu Address: 192.249.249.3 *** terminator.movie.edu can’t find carrie.movie.edu.: Query refused > ls movie.edu — This attempts a zone transfer [terminator.movie.edu] *** Can’t list domain movie.edu: Query refused >
% nslookup *** Can’t find server name for address 192.249.249.3: No response from server Default Server: wormhole.movie.edu Address: 192.249.249.1
% nslookup -d2 ———— SendRequest(), len 44 HEADER: opcode = QUERY, rcode = NOERROR header flags: query, want recursion questions = 1, answers = 0, authority records = 0, additional = 0 QUESTIONS: 3.249.249.192.in-addr.arpa, type = PTR, (5 secs) timeout (10 secs) timeout (20 secs) timeout (40 secs) SendRequest failed *** Can’t find server name for address 192.249.249.3: No response from server *** Default servers are not available
As you can see by the timeouts, it took 75 seconds for nslookup to give up. Without the debugging output, you won’t see anything printed to your screen for 75 seconds; it’ll look as if nslookup has hung.
Что означает такая ошибка No Response from Application Web Server?
There was no response from the application web server for the page you requested.
Please notify the site’s webmaster and try your request again later.
Как это переводиться?
"Не отвечает приложение веб-сервера".
"Мы не получили ответа от приложения веб-сервера для запрошенной вами страницы.
Пожалуйста известите веб-мастера сайта и попробуйте снова выполнить запрос позже."