fix(storage): fix flaky storage integration tests#13253
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces state checks in BidiUploadStreamingStream to return early when the stream is in a terminal success state during finishWrite and closeStream operations. It also refactors DirectWriteService in the integration tests to scope the request builder locally within the writeObject method. The review feedback suggests extending the state checks to cover all terminal states, including failures, to prevent potential IllegalStateException during subsequent state transitions.
| if (state.getState() == State.TERMINAL_SUCCESS) { | ||
| return true; | ||
| } |
There was a problem hiding this comment.
The check only accounts for State.TERMINAL_SUCCESS. If the stream has reached a terminal failure state (e.g., TERMINAL_FAILURE), subsequent calls to finishWrite might still trigger an IllegalStateException when calling state.isFinalizing() or other methods that enforce state transitions. Consider checking for any terminal state to ensure the method is robust against all completed stream states.
| if (state.getState() == State.TERMINAL_SUCCESS) { | ||
| return true; | ||
| } |
This PR addresses intermittent flakiness, connection refused errors, and state validation exceptions in the Google Cloud Storage integration and unit test suites under highly concurrent CI (Kokoro) environments.
Key Fixes
Testbench Lifecycle Isolation (Forks Race Condition):
<forkCount>1C</forkCount>) collided on hardcoded ports (9000/9005) and container names. When a fork completed first, it prematurely killed the shared Testbench container, causing all remaining active tests in other forks to fail withConnectException: Connection refused.fork-<port>) inRegistry.java. This fully isolates each parallel JVM process's testbench container instance.Mock Concurrency Isolation (
DirectWriteService):DirectWriteServiceinITGapicUnbufferedWritableByteChannelTestheldrequestsin a shared mutable class field. Under concurrent gRPC threads in CI, the request builder got corrupted, throwing false-positivePERMISSION_DENIED: Unexpected request chainerrors.DirectWriteServiceto keep the request builder completely isolated and local to each stream observer instance.Graceful Terminal Success Handling:
buffered.close()flushes the final bytes, successfully transitioning the stream's state toTERMINAL_SUCCESS. Immediately after,unbuffered.close()runs and callsfinishWriteagain. Querying the terminal state threwIllegalStateException: state mismatch.finishWriteandcloseStreaminsideBidiUploadStreamingStream.javato returntrue(safe no-op) immediately if the upload state has already successfully reachedTERMINAL_SUCCESS. Also updated mock classes inBidiUploadTestto support the new state checks.