Cache DownloadableArtifacts with GeneratorEngine
#34
Conversation
In subsequent PRs I'd like to introduce a SQLite-backed cache for storing hashes of generated artifacts. That will help us in avoiding redundant work and will make generator significantly quicker for re-runs that change only a few arguments.
| let isPrebuilt: Bool | ||
| } | ||
|
|
||
| let hostSwift: Item | ||
| private(set) var hostLLVM: Item | ||
| let targetSwift: Item | ||
|
|
||
| let allItems: [Item] | ||
| private let shouldUseDocker: Bool | ||
| var allItems: [Item] { |
There was a problem hiding this comment.
Fixing a bug here, where allItems wasn't recomputed after running useLLVMSources.
| do { | ||
| try await generator.generateBundle(shouldGenerateFromScratch: !self.incremental) | ||
| try await generator.shutDown() | ||
| } catch { | ||
| try await generator.shutDown() | ||
| throw error | ||
| } |
There was a problem hiding this comment.
It would be nice add a withGenerator wrapper which could take care of shutting down and rethrowing if necessary.
| private let knownLLVMBinariesVersions: [ArtifactOS: [String: CPUMapping]] = [ | ||
| .macOS: [ | ||
| "15.0.7": [ | ||
| Triple.CPU.arm64: "867c6afd41158c132ef05a8f1ddaecf476a26b91c85def8e124414f9a9ba188d", |
There was a problem hiding this comment.
A side effect of using these hashes was that we also verified that the upstream tarballs hadn't changed from the known versions. LLVM signs its binaries using GPG, so in future we could have an option to verify those signatures. This would require the user to download and validate LLVM's public key.
There was a problem hiding this comment.
I agree, checking signatures is the right way to do it. But code on main that handled checksums was inconsistent and didn't warn users on checksums mismatch. It would only redownload new archives from remote URLs on a mismatch or if a checksum was absent. From that perspective, we didn't regress, and the caching behaviour is actually improved.
MaxDesiatov
force-pushed
the
maxd/engine-downloads
branch
from
3be7fe9 to
6ca73f1
Compare
|
@swift-ci test |
|
@swift-ci test |
|
@swift-ci test |
|
@swift-ci test |
Depends on #31.
New
GeneratorEngineallows us to cache all items inDownloadableArtifactsin a generic way and to allow redundant re-downloads for artifacts that don't have their checksums hardcoded (e.g. LLVM sources forlld).This now allows us to delete all hardcoded checksums and also to cache other artifacts in the future, like
lldbinary and even whole artifact bundles.