Lars Ericson

525 request errors in API

I'm getting 525 errors intermittently on different uploads, for example

POST
{'external_prediction_set': {'external_predictions_attributes': [{'answer_id': 8546,
'value': 0.11973172672828676},
{'answer_id': 8547,
'value': 0.0732515480161981},
{'answer_id': 8548,
'value': 0.1539183295505778},
{'answer_id': 8549,
'value': 0.23715590826692748},
{'answer_id': 8550,
'value': 0.41594248743801}],
'external_predictor_attributes': {'method_name': 'm05'},
'question_id': 3053}}
REQUEST
<Response [525]>

This is supposed to be some kind of CloudFlare SSL Handshake Failed whosiwhatsit: https://community.cloudflare.com/t/community-tip-fixing-error-525-ssl-handshake-failed/44256

Any help would be appreciated.

10 Replies

Ben Roesch
Hey Lars-

Digging into this a bit. Are you still seeing these or have they stopped occurring at this point?

Thanks,
Ben
Lars Ericson
Still get them now and then. I don't think they explain my Leaderboard position though, which ought to be higher. I could have been a contender!
Attachments
EagleWings
Our submission was also cut off from time to time, it happened several times in the past week or two.
Ben Roesch
Hey all-

We made an adjustment to our Cloudflare setup that should hopefully eliminate these. Can you let me know in this thread if you continue to see them?

Thanks,
Ben
EagleWings
Our submission was cut off again at about 13:11pm today (9/6/2019)
Lars Ericson
I got unrecoverable unreliable behavior today, after uploading the 25th method (they are all the same after a certain point, prior to 25, so it's not the content):

65%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 26/40 [18:15<09:32, 40.88s/it]Traceback (most recent call last): | 21/164 [00:05<00:34, 4.18it/s]
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\contrib\pyopenssl.py", line 456, in wrap_socket
cnx.do_handshake()
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\OpenSSL\SSL.py", line 1915, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\OpenSSL\SSL.py", line 1640, in _raise_ssl_error
raise SysCallError(-1, "Unexpected EOF")
OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
conn.connect()
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\connection.py", line 344, in connect
ssl_context=context)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\util\ssl_.py", line 347, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\contrib\pyopenssl.py", line 462, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\util\retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.iarpagfchallenge.com', port=443): Max retries exceeded with url: /api/v1/external_prediction_sets (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "forecast.py", line 71, in <module>
submit_forecast_multiple(method,qid,fcast)
File "c:\Users\erxnm\Desktop\gfc2\submit_forecast_multiple.py", line 28, in submit_forecast_multiple
r=requests.post(url, headers=token, json=post)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.iarpagfchallenge.com', port=443): Max retries exceeded with url: /api/v1/external_prediction_sets (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')")))
Modified on Sept. 10, 2019, 8:13 p.m. PDT
Lars Ericson
I tried again this morning and the API is definitely not feeling well. I got one method uploaded and then it died 27 questions into the 2nd method:

9%|#########8| 162/164 [00:47<00:00, 2.30it/s]
99%|#########9| 163/164 [00:48<00:00, 2.14it/s]
5%|5 | 2/40 [01:36<30:27, 48.09s/it]
0%| | 0/164 [00:00<?, ?it/s]
1%| | 1/164 [00:00<00:53, 3.05it/s]
1%|1 | 2/164 [00:00<00:57, 2.81it/s]
2%|1 | 3/164 [00:01<00:59, 2.73it/s]
2%|2 | 4/164 [00:01<01:00, 2.66it/s]
3%|3 | 5/164 [00:01<01:03, 2.51it/s]
4%|3 | 6/164 [00:02<00:56, 2.79it/s]
4%|4 | 7/164 [00:02<00:57, 2.74it/s]
5%|4 | 8/164 [00:02<00:50, 3.12it/s]
5%|5 | 9/164 [00:03<00:53, 2.89it/s]
6%|6 | 10/164 [00:03<00:52, 2.93it/s]
7%|6 | 11/164 [00:03<00:53, 2.89it/s]
7%|7 | 12/164 [00:04<00:51, 2.93it/s]
8%|7 | 13/164 [00:04<00:55, 2.70it/s]
9%|8 | 14/164 [00:05<00:58, 2.56it/s]
9%|9 | 15/164 [00:05<01:05, 2.26it/s]
10%|9 | 16/164 [00:05<00:57, 2.57it/s]
10%|# | 17/164 [00:06<01:06, 2.21it/s]
11%|# | 18/164 [00:06<01:02, 2.33it/s]
12%|#1 | 19/164 [00:07<00:56, 2.57it/s]
12%|#2 | 20/164 [00:07<01:00, 2.39it/s]
13%|#2 | 21/164 [00:08<00:53, 2.66it/s]
13%|#3 | 22/164 [00:08<00:50, 2.80it/s]
14%|#4 | 23/164 [00:09<01:11, 1.97it/s]
15%|#4 | 24/164 [00:09<01:10, 2.00it/s]
15%|#5 | 25/164 [00:09<00:57, 2.40it/s]
16%|#5 | 26/164 [00:10<00:51, 2.66it/s]
16%|#6 | 27/164 [00:10<01:05, 2.11it/s]Traceback (most recent call last):
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\contrib\pyopenssl.py", line 456, in wrap_socket
cnx.do_handshake()
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\OpenSSL\SSL.py", line 1915, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\OpenSSL\SSL.py", line 1640, in _raise_ssl_error
raise SysCallError(-1, "Unexpected EOF")
OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
conn.connect()
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\connection.py", line 344, in connect
ssl_context=context)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\util\ssl_.py", line 347, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\contrib\pyopenssl.py", line 462, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\urllib3\util\retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.iarpagfchallenge.com', port=443): Max retries exceeded with url: /api/v1/external_prediction_sets (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "forecast.py", line 69, in <module>
submit_forecast(method, qid, fcast)
File "c:\Users\erxnm\Desktop\gfc2\submit_forecast.py", line 7, in submit_forecast
submit_prediction(method_name, question_id, answer_id, prob)
File "c:\Users\erxnm\Desktop\gfc2\submit_prediction.py", line 25, in submit_prediction
r=requests.post(url, headers=token, json=post)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Users\erxnm\Anaconda3\envs\cuda_torch\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.iarpagfchallenge.com', port=443): Max retries exceeded with url: /api/v1/external_prediction_sets (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')")))
Lars Ericson
Note, it takes at least half an hour now to upload all forecasts for 40 methods and 164 questions. We could maybe come up with a more compact format, since it should take a lot less time than that. For example if every question had 5 branches, we're really just talking about sending 154KB or 6*40*164*4, which should take half a second.
Ben Roesch
Thanks Lars -- the traces are helpful. The errors are happening at our hosting provider before the requests reach our application, so I've filed a ticket with them to ask them to look at it.

In the meantime, it looks like python's requests package does not automatically retry POST requests. Are you manually rescuing from the exception and retrying the submission? Or is the "max retries" referenced in your trace == 0?
Lars Ericson
I wait a couple seconds and then try again up to 5 or 10 tries.

But to repeat: If we had some kind of compact JSON format, this should all take half a second, not half an hour.
Let these people know about your message