Failing Kill_running_test

[jyameo]

There are new test failures on Allow (e as dynamic).foo() in avoid_dynamic_calls...[linter] Exclude ElementLocation from analyzer_use_new_elements + update migration file.

The tests

pkg/vm_service/test/kill_running_test RuntimeError (expected Pass)

are failing on configurations

vm-aot-linux-debug-x64

from log:

>testee>out> The Dart VM service is listening on http://127.0.0.1:42347/
>testee>out> The Dart DevTools debugger and profiler is available at: http://127.0.0.1:42347/devtools/?uri=ws://127.0.0.1:42347/ws
Done loading VM

out.aotsnapshot [1/1]

>testee>err> isolate terminated by Kill service request
00:01 �[32m+0�[0m�[31m -1�[0m: out.aotsnapshot �[1m�[31m[E]�[0m�[0m

  kill: (-32000) Service connection disposed

  package:vm_service/src/vm_service.dart 265       new _OutstandingRequest
  package:vm_service/src/vm_service.dart 1921      VmService._call.<fn>
  package:vm_service/src/vm_service.dart 1933      VmService._call
  package:vm_service/src/vm_service.dart 1472      VmService.kill
  pkg/vm_service/test/kill_running_test.dart 18    tests.<fn>
  pkg/vm_service/test/common/test_helper.dart 331  _ServiceTesterRunner.run.<fn>
  package:test_api/src/backend/declarer.dart 229   Declarer.test.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 227   Declarer.test.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/invoker.dart 258    Invoker._waitForOutstandingCallbacks.<fn>
  

All service tests completed successfully.

** Killing script

00:01 �[32m+0�[0m�[31m -1�[0m: �[31mSome tests failed.�[0m



Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.

stderr:
Unhandled exception:
Dummy exception to set exit code.

--- Re-run this test:
python3 tools/test.py -n vm-aot-linux-debug-x64 pkg/vm_service/test/kill_running_test

It seems there is previously been a similar failure. See #53373

[jyameo]

CC @bkonyi @derekxu16

[dart-github-bot]

Summary: The kill_running_test in the vm_service package is failing on the vm-aot-linux-debug-x64 configuration. The test is attempting to kill a running isolate using the VM service, but the isolate is terminating with an error related to the service connection being disposed. This issue may be related to a previous issue (#53373).

[alexmarkov]

I still see this test failing on the try-bots from time to time (it is flaky):

/===================================================================================\
| pkg/vm_service/test/kill_running_test broke (Pass -> RuntimeError, expected Pass) |
\===================================================================================/

--- Command "vm_compile_to_kernel" (took 25.000986s):
DART_CONFIGURATION=DebugX64 /b/s/w/ir/pkg/vm/tool/gen_kernel --aot --platform=out/DebugX64/vm_platform_strong.dill -o /b/s/w/ir/out/DebugX64/generated_compilations/vm-aot-linux-debug-x64/pkg_vm_service_test_kill_running_test/out.dill /b/s/w/ir/pkg/vm_service/test/kill_running_test.dart -Dtest_runner.configuration=vm-aot-linux-debug-x64 --packages=/b/s/w/ir/.dart_tool/package_config.json -Ddart.vm.product=false --sound-null-safety

exit code:
0

--- Command "precompiler" (took 13.000838s):
DART_CONFIGURATION=DebugX64 out/DebugX64/gen_snapshot --snapshot-kind=app-aot-assembly --assembly=/b/s/w/ir/out/DebugX64/generated_compilations/vm-aot-linux-debug-x64/pkg_vm_service_test_kill_running_test/out.S --sound-null-safety -Dtest_runner.configuration=vm-aot-linux-debug-x64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json /b/s/w/ir/out/DebugX64/generated_compilations/vm-aot-linux-debug-x64/pkg_vm_service_test_kill_running_test/out.dill

exit code:
0

--- Command "remove_kernel_file" (took 19ms):
DART_CONFIGURATION=DebugX64 rm /b/s/w/ir/out/DebugX64/generated_compilations/vm-aot-linux-debug-x64/pkg_vm_service_test_kill_running_test/out.dill

exit code:
0

--- Command "assemble" (took 903ms):
DART_CONFIGURATION=DebugX64 gcc -Wl,--no-undefined -shared -o /b/s/w/ir/out/DebugX64/generated_compilations/vm-aot-linux-debug-x64/pkg_vm_service_test_kill_running_test/out.aotsnapshot /b/s/w/ir/out/DebugX64/generated_compilations/vm-aot-linux-debug-x64/pkg_vm_service_test_kill_running_test/out.S

exit code:
0

--- Command "remove_assembly" (took 21ms):
DART_CONFIGURATION=DebugX64 rm /b/s/w/ir/out/DebugX64/generated_compilations/vm-aot-linux-debug-x64/pkg_vm_service_test_kill_running_test/out.S

exit code:
0

--- Command "vm" (took 03.000268s):
DART_CONFIGURATION=DebugX64 TEST_COMPILATION_DIR=/b/s/w/ir/out/DebugX64/generated_compilations/vm-aot-linux-debug-x64/pkg_vm_service_test_kill_running_test out/DebugX64/dartaotruntime --sound-null-safety -Dtest_runner.configuration=vm-aot-linux-debug-x64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json /b/s/w/ir/out/DebugX64/generated_compilations/vm-aot-linux-debug-x64/pkg_vm_service_test_kill_running_test/out.aotsnapshot

exit code:
255

stdout:
00:00 �[32m+0�[0m: out.aotsnapshot�[0m

** Launching SERVICE_TEST_TESTEE=true out/DebugX64/dartaotruntime --disable-service-auth-codes --profiler --sound-null-safety -Dtest_runner.configuration=vm-aot-linux-debug-x64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json --enable-vm-service:0 /b/s/w/ir/out/DebugX64/generated_compilations/vm-aot-linux-debug-x64/pkg_vm_service_test_kill_running_test/out.aotsnapshot

>testee>out> 
>testee>out> 1
** Signaled to run test queries on http://127.0.0.1:36945//

>testee>out> The Dart VM service is listening on http://127.0.0.1:36945//
>testee>out> The Dart DevTools debugger and profiler is available at: http://127.0.0.1:36945/devtools/?uri=ws://127.0.0.1:36945/ws
Done loading VM

out.aotsnapshot [1/1]

>testee>err> isolate terminated by Kill service request
00:03 �[32m+0�[0m�[31m -1�[0m: out.aotsnapshot �[1m�[31m[E]�[0m�[0m

  Expected: <-32010>
    Actual: <-32000>
  

  package:matcher/src/expect/expect.dart 149       fail
  package:matcher/src/expect/expect.dart 144       _expect
  package:matcher/src/expect/expect.dart 56        expect
  pkg/vm_service/test/kill_running_test.dart 25    tests.<fn>
  ===== asynchronous gap ===========================
  pkg/vm_service/test/common/test_helper.dart 331  _ServiceTesterRunner.run.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 229   Declarer.test.<fn>.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/declarer.dart 227   Declarer.test.<fn>
  ===== asynchronous gap ===========================
  package:test_api/src/backend/invoker.dart 258    Invoker._waitForOutstandingCallbacks.<fn>
  

All service tests completed successfully.

** Killing script

00:03 �[32m+0�[0m�[31m -1�[0m: �[31mSome tests failed.�[0m



Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.

stderr:
Unhandled exception:
Dummy exception to set exit code.

--- Re-run this test:
python3 tools/test.py -n vm-aot-linux-debug-x64 pkg/vm_service/test/kill_running_test

It looks like it fails with RPC error -32000, but handles -32010.

[a-siva]

//cc @jyameo

[jyameo]

I built the Dart SDK in a Cloudtop instance and ran the test multiple times. Each time, the test passed without any errors. Upon investigating the code further, these were my findings:

• Expected error code (32010): This is an RPCError code that indicates an application error.
• Actual error code (32000): This is an RPCError code that indicates a service connection was disposed.

The definitions for these RPCError codes can be found here: vm_service.dart#L2075-L2082

While I haven’t pinpointed the root cause of the flakiness, I did find some potentially suspicious code:

• vm_service.dart#L1912-L1916
• vm_service.dart#L1883-L1886

In these instances, when an error occurs, the code throws RPCErrorKind.kServerError.code (32000) but uses the message "Service connection disposed". This message is defined as the description for RPCErrorKind.kConnectionDisposed (32010). Could it be that the wrong error code is being thrown here?

cc @a-siva @alexmarkov

[bkonyi]

In these instances, when an error occurs, the code throws RPCErrorKind.kServerError.code (32000) but uses the message "Service connection disposed". This message is defined as the description for RPCErrorKind.kConnectionDisposed (32010). Could it be that the wrong error code is being thrown here?

Good catch! I think the plan was to switch those instances of RPCErrorKind.kServerError with RPCErrorKind.kConnectionDisposed (see the description e099a00) once we gave clients the opportunity to migrate to handle the new error code. If we update this test to allow for either error code, this should stop flaking.