*: split up large packages to speed up builds

[irfansharif]

Is your feature request related to a problem? Please describe.

This is a catch-all issue to split up our largest packages (as measured by build time) to increase build parallelism. Bazel can spit out how much time is spent compiling each package, and highlights what the bottleneck packages are.

$ dev build short -- --profile=short.gz # load through chrome, browsing to chrome://tracing
$ bazel analyze-profile short.gz # textual analysis

Describe the solution you'd like

The following packages are common offendors. There are probably more.

• pkg/sql
• pkg/sql/parser
• pkg/sql/colexec/colexecproj

   24.614 s    6.16%   action 'GoCompilePkg pkg/sql/colexec/colexecproj/colexecproj.a'
   ...
    9.509 s    2.38%   action 'GoCompilePkg pkg/sql/sql.a'

Related: #60547, #78237.

Jira issue: CRDB-14743

[irfansharif]

Here's a representative profile: local.gz.

Critical path (75.991 s):
       Time Percentage   Description
    3.613 s    4.75%   action 'GoCompilePkg pkg/roachpb/roachpb.a'
     894 ms    1.18%   action 'GoCompilePkg pkg/clusterversion/clusterversion.a'
     245 ms    0.32%   action 'GoCompilePkg pkg/settings/cluster/cluster.a'
    0.32 ms    0.00%   action 'GoCompilePkg pkg/security/security.a'
     652 ms    0.86%   action 'GoCompilePkg pkg/base/base.a'
     784 ms    1.03%   action 'GoCompilePkg pkg/util/admission/admission.a'
    0.64 ms    0.00%   action 'GoCompilePkg pkg/sql/sessiondatapb/sessiondatapb.a'
     749 ms    0.99%   action 'GoCompilePkg pkg/kv/kv.a'
    2.844 s    3.74%   action 'GoCompilePkg pkg/sql/sem/tree/tree.a'
    5.340 s    7.03%   action 'GoCompilePkg pkg/sql/parser/parser.a'
     888 ms    1.17%   action 'GoCompilePkg pkg/sql/catalog/descpb/descpb.a'
     985 ms    1.30%   action 'GoCompilePkg pkg/sql/catalog/catalog.a'
    1.178 s    1.55%   action 'GoCompilePkg pkg/sql/catalog/multiregion/multiregion.a'
    1.701 s    2.24%   action 'GoCompilePkg pkg/sql/catalog/typedesc/typedesc.a'
    1.445 s    1.90%   action 'GoCompilePkg pkg/sql/catalog/tabledesc/tabledesc.a'
    1.012 s    1.33%   action 'GoCompilePkg pkg/sql/catalog/systemschema/systemschema.a'
     941 ms    1.24%   action 'GoCompilePkg pkg/config/config.a'
    1.278 s    1.68%   action 'GoCompilePkg pkg/gossip/gossip.a'
    1.222 s    1.61%   action 'GoCompilePkg pkg/kv/kvclient/kvcoord/kvcoord.a'
     646 ms    0.85%   action 'GoCompilePkg pkg/kv/kvclient/rangefeed/rangefeed.a'
     723 ms    0.95%   action 'GoCompilePkg pkg/sql/catalog/lease/lease.a'
     703 ms    0.92%   action 'GoCompilePkg pkg/sql/catalog/hydratedtables/hydratedtables.a'
     667 ms    0.88%   action 'GoCompilePkg pkg/sql/catalog/descs/descs.a'
    3.190 s    4.20%   action 'GoCompilePkg pkg/sql/sem/builtins/builtins.a'
     707 ms    0.93%   action 'GoCompilePkg pkg/jobs/jobs.a'
    1.187 s    1.56%   action 'GoCompilePkg pkg/sql/row/row.a'
     778 ms    1.02%   action 'GoCompilePkg pkg/sql/execinfra/execinfra.a'
     694 ms    0.91%   action 'GoCompilePkg pkg/sql/colcontainer/colcontainer.a'
     356 ms    0.47%   action 'GoCompilePkg pkg/sql/colexec/colexecutils/colexecutils.a'
    1.686 s    2.22%   action 'GoCompilePkg pkg/sql/colexec/colexecbase/colexecbase.a'
    7.009 s    9.22%   action 'GoCompilePkg pkg/sql/colexec/colexecproj/colexecproj.a'
     428 ms    0.56%   action 'GoCompilePkg pkg/sql/colexec/colbuilder/colbuilder.a'
     473 ms    0.62%   action 'GoCompilePkg pkg/sql/colflow/colflow.a'
     384 ms    0.51%   action 'GoCompilePkg pkg/sql/distsql/distsql.a'
    5.841 s    7.69%   action 'GoCompilePkg pkg/sql/sql.a'
    1.519 s    2.00%   action 'GoCompilePkg pkg/sql/gcjob/gcjob.a'
    1.016 s    1.34%   action 'GoCompilePkg pkg/sql/importer/importer.a'
    2.301 s    3.03%   action 'GoCompilePkg pkg/server/server.a'
    1.293 s    1.70%   action 'GoCompilePkg pkg/cli/democluster/democluster.a'
    2.071 s    2.73%   action 'GoCompilePkg pkg/cli/cli.a'
     310 ms    0.41%   action 'GoCompilePkg pkg/ccl/cliccl/cliccl.a'
     145 ms    0.19%   action 'GoCompilePkg pkg/ccl/ccl.a'
     187 ms    0.25%   action 'GoCompilePkg pkg/cmd/cockroach-short/cockroach-short.a'
   15.905 s   20.93%   action 'GoLink pkg/cmd/cockroach-short/cockroach-short_/cockroach-short'
    0.04 ms    0.00%   runfiles for //pkg/cmd/cockroach-short cockroach-short

The periods of low build parallelism (pink graph) correspond to roachpb, parser, colexecproj, sql, server respectively.

[rickystewart]

The 5 packages you named match up to my experience as well, I routinely see them as bottlenecks.

[yuzefovich]

Here is an updated analysis (with #79938):

Critical path (119.937 s):
       Time Percentage   Description
    5.271 s    4.40%   action 'GoCompilePkg pkg/roachpb/roachpb.a'
    2.243 s    1.87%   action 'GoCompilePkg pkg/clusterversion/clusterversion.a'
     934 ms    0.78%   action 'GoCompilePkg pkg/settings/cluster/cluster.a'
    0.57 ms    0.00%   action 'GoCompilePkg pkg/util/mon/mon.a'
     865 ms    0.72%   action 'GoCompilePkg pkg/base/base.a'
     861 ms    0.72%   action 'GoCompilePkg pkg/util/admission/admission.a'
    0.40 ms    0.00%   action 'GoCompilePkg pkg/sql/sessiondatapb/sessiondatapb.a'
     739 ms    0.62%   action 'GoCompilePkg pkg/kv/kv.a'
    1.393 s    1.16%   action 'GoCompilePkg pkg/kv/kvserver/kvserverbase/kvserverbase.a'
    4.281 s    3.57%   action 'GoCompilePkg pkg/sql/sem/tree/tree.a'
   12.780 s   10.66%   action 'GoCompilePkg pkg/sql/parser/parser.a'
    1.347 s    1.12%   action 'GoCompilePkg pkg/sql/catalog/descpb/descpb.a'
    1.639 s    1.37%   action 'GoCompilePkg pkg/sql/catalog/catalog.a'
    1.118 s    0.93%   action 'GoCompilePkg pkg/sql/catalog/multiregion/multiregion.a'
    1.807 s    1.51%   action 'GoCompilePkg pkg/sql/catalog/typedesc/typedesc.a'
    2.529 s    2.11%   action 'GoCompilePkg pkg/sql/catalog/tabledesc/tabledesc.a'
    1.557 s    1.30%   action 'GoCompilePkg pkg/sql/catalog/systemschema/systemschema.a'
    1.494 s    1.25%   action 'GoCompilePkg pkg/config/config.a'
    2.816 s    2.35%   action 'GoCompilePkg pkg/gossip/gossip.a'
    2.653 s    2.21%   action 'GoCompilePkg pkg/kv/kvclient/kvcoord/kvcoord.a'
    1.025 s    0.85%   action 'GoCompilePkg pkg/kv/kvclient/rangefeed/rangefeed.a'
    1.726 s    1.44%   action 'GoCompilePkg pkg/sql/catalog/lease/lease.a'
    1.046 s    0.87%   action 'GoCompilePkg pkg/sql/catalog/hydratedtables/hydratedtables.a'
    1.182 s    0.99%   action 'GoCompilePkg pkg/sql/catalog/descs/descs.a'
    5.816 s    4.85%   action 'GoCompilePkg pkg/sql/sem/builtins/builtins.a'
    5.345 s    4.46%   action 'GoCompilePkg pkg/sql/opt/memo/memo.a'
    3.320 s    2.77%   action 'GoCompilePkg pkg/sql/opt/norm/norm.a'
     739 ms    0.62%   action 'GoCompilePkg pkg/sql/opt/idxconstraint/idxconstraint.a'
     487 ms    0.41%   action 'GoCompilePkg pkg/sql/opt/invertedidx/invertedidx.a'
    1.307 s    1.09%   action 'GoCompilePkg pkg/sql/opt/xform/xform.a'
     553 ms    0.46%   action 'GoCompilePkg pkg/sql/opt/optgen/exprgen/exprgen.a'
    1.557 s    1.30%   action 'GoCompilePkg pkg/sql/opt/optbuilder/optbuilder.a'
    8.242 s    6.87%   action 'GoCompilePkg pkg/sql/sql.a'
    2.610 s    2.18%   action 'GoCompilePkg pkg/sql/gcjob/gcjob.a'
    1.907 s    1.59%   action 'GoCompilePkg pkg/sql/importer/importer.a'
    3.585 s    2.99%   action 'GoCompilePkg pkg/server/server.a'
    1.933 s    1.61%   action 'GoCompilePkg pkg/testutils/testcluster/testcluster.a'
    2.730 s    2.28%   action 'GoCompilePkg pkg/ccl/backupccl/backupccl.a'
     718 ms    0.60%   action 'GoCompilePkg pkg/ccl/streamingccl/streamingest/streamingest.a'
     240 ms    0.20%   action 'GoCompilePkg pkg/ccl/ccl.a'
     226 ms    0.19%   action 'GoCompilePkg pkg/cmd/cockroach-short/cockroach-short.a'
   27.314 s   22.77%   action 'GoLink pkg/cmd/cockroach-short/cockroach-short_/cockroach-short'
    0.11 ms    0.00%   runfiles for //pkg/cmd/cockroach-short cockroach-short

Packages of the vectorized engine are no longer on the critical path (at least in this run), woohoo!