[NFCI][WPD]Use unique string saver to store type id #106932
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
|
@llvm/pr-subscribers-llvm-ir Author: Mingming Liu (minglotus-6) ChangesCurrently, both TypeIdMap and TypeIdCompatibleVtableMap keep type-id as With this change, index uses a unique-string-saver to own the string copies and two maps above can use string references to save some memory. This shows a 3% memory reduction (from 8.2GiB to 7.9GiB) in an internal binary with high indexing memory usage. Full diff: https://github.com/llvm/llvm-project/pull/106932.diff 3 Files Affected:
diff --git a/llvm/include/llvm/IR/ModuleSummaryIndex.h b/llvm/include/llvm/IR/ModuleSummaryIndex.h
index 39c60229aa1d81..b32aebb25cf5c7 100644
--- a/llvm/include/llvm/IR/ModuleSummaryIndex.h
+++ b/llvm/include/llvm/IR/ModuleSummaryIndex.h
@@ -1315,7 +1315,7 @@ using GVSummaryPtrSet = std::unordered_set<GlobalValueSummary *>;
/// Map of a type GUID to type id string and summary (multimap used
/// in case of GUID conflicts).
using TypeIdSummaryMapTy =
- std::multimap<GlobalValue::GUID, std::pair<std::string, TypeIdSummary>>;
+ std::multimap<GlobalValue::GUID, std::pair<StringRef, TypeIdSummary>>;
/// The following data structures summarize type metadata information.
/// For type metadata overview see https://llvm.org/docs/TypeMetadata.html.
@@ -1351,6 +1351,9 @@ class ModuleSummaryIndex {
/// Holds strings for combined index, mapping to the corresponding module ID.
ModulePathStringTableTy ModulePathStringTable;
+ BumpPtrAllocator TypeIdSaverAlloc;
+ UniqueStringSaver TypeIdSaver;
+
/// Mapping from type identifier GUIDs to type identifier and its summary
/// information. Produced by thin link.
TypeIdSummaryMapTy TypeIdMap;
@@ -1359,7 +1362,7 @@ class ModuleSummaryIndex {
/// with that type identifier's metadata. Produced by per module summary
/// analysis and consumed by thin link. For more information, see description
/// above where TypeIdCompatibleVtableInfo is defined.
- std::map<std::string, TypeIdCompatibleVtableInfo, std::less<>>
+ std::map<StringRef, TypeIdCompatibleVtableInfo, std::less<>>
TypeIdCompatibleVtableMap;
/// Mapping from original ID to GUID. If original ID can map to multiple
@@ -1455,8 +1458,8 @@ class ModuleSummaryIndex {
// See HaveGVs variable comment.
ModuleSummaryIndex(bool HaveGVs, bool EnableSplitLTOUnit = false,
bool UnifiedLTO = false)
- : HaveGVs(HaveGVs), EnableSplitLTOUnit(EnableSplitLTOUnit),
- UnifiedLTO(UnifiedLTO), Saver(Alloc) {}
+ : TypeIdSaver(TypeIdSaverAlloc), HaveGVs(HaveGVs), EnableSplitLTOUnit(EnableSplitLTOUnit),
+ UnifiedLTO(UnifiedLTO), Saver(Alloc) {}
// Current version for the module summary in bitcode files.
// The BitcodeSummaryVersion should be bumped whenever we introduce changes
@@ -1829,8 +1832,8 @@ class ModuleSummaryIndex {
for (auto &[GUID, TypeIdPair] : make_range(TidIter))
if (TypeIdPair.first == TypeId)
return TypeIdPair.second;
- auto It = TypeIdMap.insert(
- {GlobalValue::getGUID(TypeId), {std::string(TypeId), TypeIdSummary()}});
+ auto It = TypeIdMap.insert({GlobalValue::getGUID(TypeId),
+ {TypeIdSaver.save(TypeId), TypeIdSummary()}});
return It->second.second;
}
@@ -1859,7 +1862,7 @@ class ModuleSummaryIndex {
/// the ThinLTO backends.
TypeIdCompatibleVtableInfo &
getOrInsertTypeIdCompatibleVtableSummary(StringRef TypeId) {
- return TypeIdCompatibleVtableMap[std::string(TypeId)];
+ return TypeIdCompatibleVtableMap[TypeIdSaver.save(TypeId)];
}
/// For the given \p TypeId, this returns the TypeIdCompatibleVtableMap
diff --git a/llvm/include/llvm/IR/ModuleSummaryIndexYAML.h b/llvm/include/llvm/IR/ModuleSummaryIndexYAML.h
index 7c405025630c95..f1d3c437cdfca8 100644
--- a/llvm/include/llvm/IR/ModuleSummaryIndexYAML.h
+++ b/llvm/include/llvm/IR/ModuleSummaryIndexYAML.h
@@ -313,11 +313,11 @@ template <> struct CustomMappingTraits<TypeIdSummaryMapTy> {
static void inputOne(IO &io, StringRef Key, TypeIdSummaryMapTy &V) {
TypeIdSummary TId;
io.mapRequired(Key.str().c_str(), TId);
- V.insert({GlobalValue::getGUID(Key), {std::string(Key), TId}});
+ V.insert({GlobalValue::getGUID(Key), {Key, TId}});
}
static void output(IO &io, TypeIdSummaryMapTy &V) {
for (auto &TidIter : V)
- io.mapRequired(TidIter.second.first.c_str(), TidIter.second.second);
+ io.mapRequired(TidIter.second.first.str().c_str(), TidIter.second.second);
}
};
@@ -327,7 +327,20 @@ template <> struct MappingTraits<ModuleSummaryIndex> {
if (!io.outputting())
CustomMappingTraits<GlobalValueSummaryMapTy>::fixAliaseeLinks(
index.GlobalValueMap);
- io.mapOptional("TypeIdMap", index.TypeIdMap);
+
+ if (io.outputting()) {
+ io.mapOptional("TypeIdMap", index.TypeIdMap);
+ } else {
+ TypeIdSummaryMapTy TypeIdMap;
+ io.mapOptional("TypeIdMap", TypeIdMap);
+ for (auto &[TypeGUID, TypeIdSummaryMap] : TypeIdMap) {
+ // Save type id references in index and point TypeIdMap to use the
+ // references owned by index.
+ StringRef KeyRef = index.TypeIdSaver.save(TypeIdSummaryMap.first);
+ index.TypeIdMap.insert({TypeGUID, {KeyRef, TypeIdSummaryMap.second}});
+ }
+ }
+
io.mapOptional("WithGlobalValueDeadStripping",
index.WithGlobalValueDeadStripping);
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 24a4c2e8303d5a..6f50930a0fc384 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -4165,7 +4165,7 @@ static void writeWholeProgramDevirtResolution(
static void writeTypeIdSummaryRecord(SmallVector<uint64_t, 64> &NameVals,
StringTableBuilder &StrtabBuilder,
- const std::string &Id,
+ StringRef Id,
const TypeIdSummary &Summary) {
NameVals.push_back(StrtabBuilder.add(Id));
NameVals.push_back(Id.size());
@@ -4184,7 +4184,7 @@ static void writeTypeIdSummaryRecord(SmallVector<uint64_t, 64> &NameVals,
static void writeTypeIdCompatibleVtableSummaryRecord(
SmallVector<uint64_t, 64> &NameVals, StringTableBuilder &StrtabBuilder,
- const std::string &Id, const TypeIdCompatibleVtableInfo &Summary,
+ StringRef Id, const TypeIdCompatibleVtableInfo &Summary,
ValueEnumerator &VE) {
NameVals.push_back(StrtabBuilder.add(Id));
NameVals.push_back(Id.size());
|
| // Save type id references in index and point TypeIdMap to use the | ||
| // references owned by index. | ||
| StringRef KeyRef = index.TypeIdSaver.save(TypeIdSummaryMap.first); | ||
| index.TypeIdMap.insert({TypeGUID, {KeyRef, TypeIdSummaryMap.second}}); |
There was a problem hiding this comment.
Instead of doing all this can you instead change CustomMappingTraits to invoke index.getOrInsertTypeIdSummary?
There was a problem hiding this comment.
Before I figured out this way, I tried to change CustomMappingTraits. If I read correctly, the input/output methods don't take 'index' as a parameter, and 'TypeIdMap' doesn't have a pointer to 'index' either.
I updated the PR to 'move' TypeIdSummaryMap.second in the latest reply. yaml IO is likely not on the critical path, but std::move should be safe to invoke here.
There was a problem hiding this comment.
ah. ok yeah I missed the fact that you don't have the index there.
teresajohnson
approved these changes
Nov 20, 2024
|
The failed tests are all under clang/test while this PR changes llvm. Test failures shouldn't be related. |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/30/builds/10709 Here is the relevant piece of the build log for the reference |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, both TypeIdMap and TypeIdCompatibleVtableMap keep type-id as
std::stringin the combined index for LTO indexing analysis.With this change, index uses a unique-string-saver to own the string copies and two maps above can use string references to save some memory.
This shows a 3% memory reduction (from 8.2GiB to 7.9GiB) in an internal binary with high indexing memory usage.