Remove dependency on builder since these workflows are moved to test-infra

Remove dependency on builder since these workflows are moved to test-infra #1066