Received message larger than max (4237525 vs. 4194304) - Python client cannot list experiments #849
Description
After recording ~130 experiments, attempting to list them from the Python client fails with an oversized message exception. Adding a filter does not work around the issue.
Traceback when calling keepsake.experiments.list():
_InactiveRpcError Traceback (most recent call last)
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in wrapped(*args, **kwargs)
31 try:
---> 32 return f(*args, **kwargs)
33 except grpc.RpcError as e:
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in list_experiments(self)
208 def list_experiments(self) -> List[Experiment]:
--> 209 ret = self.stub.ListExperiments(pb.ListExperimentsRequest())
210 return pb_convert.experiments_from_pb(self.project, ret.experiments)
~/auto-stop-tar/.venv/lib/python3.9/site-packages/grpc/_channel.py in call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
945 wait_for_ready, compression)
--> 946 return _end_unary_response_blocking(state, call, False, None)
947
~/auto-stop-tar/.venv/lib/python3.9/site-packages/grpc/_channel.py in _end_unary_response_blocking(state, call, with_call, deadline)
848 else:
--> 849 raise _InactiveRpcError(state)
850
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "Received message larger than max (4237525 vs. 4194304)"
debug_error_string = "{"created":"@1645296119.443456300","description":"Received message larger than max (4237525 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":205,"grpc_status":8}"
During handling of the above exception, another exception occurred:
Exception Traceback (most recent call last)
/tmp/ipykernel_4297/1455666872.py in
----> 1 experiments = keepsake.experiments.list(lambda e: "autotar" in e.params["run_group"])
2 len(experiments)
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/experiment.py in list(self, filter)
387 Return all experiments for a project, sorted by creation date.
388 """
--> 389 experiments = self.project._daemon().list_experiments()
390 result: ExperimentList = ExperimentList()
391 for exp in experiments:
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in wrapped(*args, **kwargs)
38 if status_code:
39 raise handle_exception(status_code, details)
---> 40 raise Exception(details)
41
42 return wrapped
Exception: Received message larger than max (4237525 vs. 4194304)