[TypeProf][InstrPGO] Introduce raw and instr profile format change for type profiling. #81691
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -94,6 +94,25 @@ INSTR_PROF_DATA(const uint32_t, llvm::Type::getInt32Ty(Ctx), NumBitmapBytes, \ | |
| #undef INSTR_PROF_DATA | ||
| /* INSTR_PROF_DATA end. */ | ||
|
|
||
| /* For a virtual table object, record the name hash to associate profiled | ||
| * addresses with global variables, and record {starting address, size in bytes} | ||
| * to map the profiled virtual table (which usually have an offset from the | ||
| * starting address) back to a virtual table object. */ | ||
| #ifndef INSTR_PROF_VTABLE_DATA | ||
| #define INSTR_PROF_VTABLE_DATA(Type, LLVMType, Name, Initializer) | ||
| #else | ||
| #define INSTR_PROF_VTABLE_DATA_DEFINED | ||
| #endif | ||
| INSTR_PROF_VTABLE_DATA(const uint64_t, llvm::Type::getInt64Ty(Ctx), \ | ||
| VTableNameHash, ConstantInt::get(llvm::Type::getInt64Ty(Ctx), \ | ||
| IndexedInstrProf::ComputeHash(PGOVTableName))) | ||
| INSTR_PROF_VTABLE_DATA(const IntPtrT, llvm::Type::getInt8PtrTy(Ctx), \ | ||
|
There was a problem hiding this comment. Diffing the contents here with the file in There was a problem hiding this comment. good catch. Fixed all the inconsistencies between two 'InstrProfData.inc`files. There was a problem hiding this comment. also sent out an nfc to clean up the use of |
||
| VTablePointer, VTableAddr) | ||
| INSTR_PROF_VTABLE_DATA(const uint32_t, llvm::Type::getInt32Ty(Ctx), VTableSize, \ | ||
| ConstantInt::get(llvm::Type::getInt32Ty(Ctx), \ | ||
| VTableSizeVal)) | ||
| #undef INSTR_PROF_VTABLE_DATA | ||
| /* INSTR_PROF_VTABLE_DATA end. */ | ||
|
|
||
| /* This is an internal data structure used by value profiler. It | ||
| * is defined here to allow serialization code sharing by LLVM | ||
|
|
@@ -145,6 +164,8 @@ INSTR_PROF_RAW_HEADER(uint64_t, CountersDelta, | |
| INSTR_PROF_RAW_HEADER(uint64_t, BitmapDelta, | ||
| (uintptr_t)BitmapBegin - (uintptr_t)DataBegin) | ||
| INSTR_PROF_RAW_HEADER(uint64_t, NamesDelta, (uintptr_t)NamesBegin) | ||
| INSTR_PROF_RAW_HEADER(uint64_t, VNamesSize, VNamesSize) | ||
| INSTR_PROF_RAW_HEADER(uint64_t, NumVTables, NumVTables) | ||
| INSTR_PROF_RAW_HEADER(uint64_t, ValueKindLast, IPVK_Last) | ||
| #undef INSTR_PROF_RAW_HEADER | ||
| /* INSTR_PROF_RAW_HEADER end */ | ||
|
|
@@ -186,13 +207,28 @@ VALUE_PROF_FUNC_PARAM(uint32_t, CounterIndex, Type::getInt32Ty(Ctx)) | |
| VALUE_PROF_KIND(IPVK_IndirectCallTarget, 0, "indirect call target") | ||
| /* For memory intrinsic functions size profiling. */ | ||
| VALUE_PROF_KIND(IPVK_MemOPSize, 1, "memory intrinsic functions size") | ||
| /* For virtual table address profiling, the address point of the virtual table | ||
| * (i.e., the address contained in objects pointing to a virtual table) are | ||
| * profiled. Note this may not be the address of the per C++ class virtual table | ||
| * object (e.g., there might be an offset). | ||
mingmingl-llvm marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| * | ||
| * The profiled addresses are stored in raw profile, together with the following | ||
| * two types of information. | ||
| * 1. The (starting and ending) addresses of per C++ class virtual table objects. | ||
| * 2. The (compressed) virtual table object names. | ||
| * RawInstrProfReader converts profiled virtual table addresses to virtual table | ||
| * objects' MD5 hash. | ||
mingmingl-llvm marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| */ | ||
| VALUE_PROF_KIND(IPVK_VTableTarget, 2, "The address of the compatible vtable (i.e., " | ||
| "there is an offset from this address to per C++ " | ||
mingmingl-llvm marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "class virtual table global variable.)") | ||
| /* These two kinds must be the last to be | ||
| * declared. This is to make sure the string | ||
| * array created with the template can be | ||
| * indexed with the kind value. | ||
| */ | ||
| VALUE_PROF_KIND(IPVK_First, IPVK_IndirectCallTarget, "first") | ||
modiking marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| VALUE_PROF_KIND(IPVK_Last, IPVK_VTableTarget, "last") | ||
|
|
||
| #undef VALUE_PROF_KIND | ||
| /* VALUE_PROF_KIND end */ | ||
|
|
@@ -282,12 +318,18 @@ INSTR_PROF_SECT_ENTRY(IPSK_bitmap, \ | |
| INSTR_PROF_SECT_ENTRY(IPSK_name, \ | ||
| INSTR_PROF_QUOTE(INSTR_PROF_NAME_COMMON), \ | ||
| INSTR_PROF_NAME_COFF, "__DATA,") | ||
| INSTR_PROF_SECT_ENTRY(IPSK_vname, \ | ||
| INSTR_PROF_QUOTE(INSTR_PROF_VNAME_COMMON), \ | ||
| INSTR_PROF_VNAME_COFF, "__DATA,") | ||
| INSTR_PROF_SECT_ENTRY(IPSK_vals, \ | ||
| INSTR_PROF_QUOTE(INSTR_PROF_VALS_COMMON), \ | ||
| INSTR_PROF_VALS_COFF, "__DATA,") | ||
| INSTR_PROF_SECT_ENTRY(IPSK_vnodes, \ | ||
| INSTR_PROF_QUOTE(INSTR_PROF_VNODES_COMMON), \ | ||
| INSTR_PROF_VNODES_COFF, "__DATA,") | ||
| INSTR_PROF_SECT_ENTRY(IPSK_vtab, \ | ||
| INSTR_PROF_QUOTE(INSTR_PROF_VTAB_COMMON), \ | ||
| INSTR_PROF_VTAB_COFF, "__DATA,") | ||
| INSTR_PROF_SECT_ENTRY(IPSK_covmap, \ | ||
| INSTR_PROF_QUOTE(INSTR_PROF_COVMAP_COMMON), \ | ||
| INSTR_PROF_COVMAP_COFF, "__LLVM_COV,") | ||
|
|
@@ -663,9 +705,9 @@ serializeValueProfDataFrom(ValueProfRecordClosure *Closure, | |
| (uint64_t)'f' << 16 | (uint64_t)'R' << 8 | (uint64_t)129 | ||
|
|
||
| /* Raw profile format version (start from 1). */ | ||
| #define INSTR_PROF_RAW_VERSION 10 | ||
| /* Indexed profile format version (start from 1). */ | ||
| #define INSTR_PROF_INDEX_VERSION 12 | ||
| /* Coverage mapping format version (start from 0). */ | ||
| #define INSTR_PROF_COVMAP_VERSION 6 | ||
|
|
||
|
|
@@ -703,10 +745,12 @@ serializeValueProfDataFrom(ValueProfRecordClosure *Closure, | |
| than WIN32 */ | ||
| #define INSTR_PROF_DATA_COMMON __llvm_prf_data | ||
| #define INSTR_PROF_NAME_COMMON __llvm_prf_names | ||
| #define INSTR_PROF_VNAME_COMMON __llvm_prf_vtabnames | ||
| #define INSTR_PROF_CNTS_COMMON __llvm_prf_cnts | ||
| #define INSTR_PROF_BITS_COMMON __llvm_prf_bits | ||
| #define INSTR_PROF_VALS_COMMON __llvm_prf_vals | ||
| #define INSTR_PROF_VNODES_COMMON __llvm_prf_vnds | ||
| #define INSTR_PROF_VTAB_COMMON __llvm_prf_vtab | ||
| #define INSTR_PROF_COVMAP_COMMON __llvm_covmap | ||
| #define INSTR_PROF_COVFUN_COMMON __llvm_covfun | ||
| #define INSTR_PROF_COVDATA_COMMON __llvm_covdata | ||
|
|
@@ -717,10 +761,12 @@ serializeValueProfDataFrom(ValueProfRecordClosure *Closure, | |
| */ | ||
| #define INSTR_PROF_DATA_COFF ".lprfd$M" | ||
| #define INSTR_PROF_NAME_COFF ".lprfn$M" | ||
| #define INSTR_PROF_VNAME_COFF ".lprfvn$M" | ||
| #define INSTR_PROF_CNTS_COFF ".lprfc$M" | ||
| #define INSTR_PROF_BITS_COFF ".lprfb$M" | ||
| #define INSTR_PROF_VALS_COFF ".lprfv$M" | ||
| #define INSTR_PROF_VNODES_COFF ".lprfnd$M" | ||
| #define INSTR_PROF_VTAB_COFF ".lprfvt$M" | ||
| #define INSTR_PROF_COVMAP_COFF ".lcovmap$M" | ||
| #define INSTR_PROF_COVFUN_COFF ".lcovfun$M" | ||
| /* Since cov data and cov names sections are not allocated, we don't need to | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -51,10 +51,14 @@ uint64_t __llvm_profile_get_size_for_buffer(void) { | |
| const char *BitmapEnd = __llvm_profile_end_bitmap(); | ||
| const char *NamesBegin = __llvm_profile_begin_names(); | ||
| const char *NamesEnd = __llvm_profile_end_names(); | ||
| const VTableProfData *VTableBegin = __llvm_profile_begin_vtables(); | ||
| const VTableProfData *VTableEnd = __llvm_profile_end_vtables(); | ||
| const char *VNamesBegin = __llvm_profile_begin_vtabnames(); | ||
| const char *VNamesEnd = __llvm_profile_end_vtabnames(); | ||
|
|
||
| return __llvm_profile_get_size_for_buffer_internal( | ||
| DataBegin, DataEnd, CountersBegin, CountersEnd, BitmapBegin, BitmapEnd, | ||
| NamesBegin, NamesEnd, VTableBegin, VTableEnd, VNamesBegin, VNamesEnd); | ||
| } | ||
|
|
||
| COMPILER_RT_VISIBILITY | ||
|
|
@@ -71,6 +75,19 @@ uint64_t __llvm_profile_get_data_size(const __llvm_profile_data *Begin, | |
| return __llvm_profile_get_num_data(Begin, End) * sizeof(__llvm_profile_data); | ||
| } | ||
|
|
||
| COMPILER_RT_VISIBILITY | ||
| uint64_t __llvm_profile_get_num_vtable(const VTableProfData *Begin, | ||
| const VTableProfData *End) { | ||
| intptr_t EndI = (intptr_t)End, BeginI = (intptr_t)Begin; | ||
| return (EndI + sizeof(VTableProfData) - 1 - BeginI) / sizeof(VTableProfData); | ||
|
There was a problem hiding this comment. I don't follow the math for the numerator. Can you add a comment to explain? There was a problem hiding this comment. Originally I implemented From offline chat, the conclusion (based on reverse reasoning) is that
From inspecting the elf sections of So it makes no difference (at least for elf) if we remove There was a problem hiding this comment. Or rather, I wonder if I shall make There was a problem hiding this comment. I think new code should not include it just because it was there before. If you update the old code and it breaks something post-commit then reverting this patch will be painful. If you want to keep things consistent, I would split out the change to the old code as a small patch and submit it soon (and monitor for any breakage - though I believe it is unlikely to happen). There was a problem hiding this comment. sounds good. I'll send out separate patches to update existing code ( |
||
| } | ||
|
|
||
| COMPILER_RT_VISIBILITY | ||
| uint64_t __llvm_profile_get_vtable_section_size(const VTableProfData *Begin, | ||
| const VTableProfData *End) { | ||
| return __llvm_profile_get_num_vtable(Begin, End) * sizeof(VTableProfData); | ||
|
There was a problem hiding this comment. Can this just be There was a problem hiding this comment. I updated both https://gcc.godbolt.org/z/MWaG6cEPn shows the generated codes before and after are equivalent (as long as |
||
| } | ||
|
|
||
| COMPILER_RT_VISIBILITY size_t __llvm_profile_counter_entry_size(void) { | ||
| if (__llvm_profile_get_version() & VARIANT_MASK_BYTE_COVERAGE) | ||
| return sizeof(uint8_t); | ||
|
|
@@ -119,21 +136,33 @@ static int needsCounterPadding(void) { | |
| } | ||
|
|
||
| COMPILER_RT_VISIBILITY | ||
| int __llvm_profile_get_padding_sizes_for_counters( | ||
| uint64_t DataSize, uint64_t CountersSize, uint64_t NumBitmapBytes, | ||
| uint64_t NamesSize, uint64_t VTableSize, uint64_t VNameSize, | ||
| uint64_t *PaddingBytesBeforeCounters, uint64_t *PaddingBytesAfterCounters, | ||
| uint64_t *PaddingBytesAfterBitmapBytes, uint64_t *PaddingBytesAfterNames, | ||
| uint64_t *PaddingBytesAfterVTable, uint64_t *PaddingBytesAfterVName) { | ||
| // Counter padding is needed only if continuous mode is enabled. | ||
| if (!needsCounterPadding()) { | ||
| *PaddingBytesBeforeCounters = 0; | ||
| *PaddingBytesAfterCounters = | ||
| __llvm_profile_get_num_padding_bytes(CountersSize); | ||
| *PaddingBytesAfterBitmapBytes = | ||
| __llvm_profile_get_num_padding_bytes(NumBitmapBytes); | ||
| *PaddingBytesAfterNames = __llvm_profile_get_num_padding_bytes(NamesSize); | ||
| if (PaddingBytesAfterVTable != NULL) | ||
| *PaddingBytesAfterVTable = | ||
| __llvm_profile_get_num_padding_bytes(VTableSize); | ||
| if (PaddingBytesAfterVName != NULL) | ||
| *PaddingBytesAfterVName = __llvm_profile_get_num_padding_bytes(VNameSize); | ||
| return 0; | ||
| } | ||
|
|
||
| // Value profiling not supported in continuous mode at profile-write time. | ||
| // Return -1 to alert the incompatibility. | ||
| if (VTableSize != 0 || VNameSize != 0) | ||
| return -1; | ||
|
|
||
| // In continuous mode, the file offsets for headers and for the start of | ||
| // counter sections need to be page-aligned. | ||
| *PaddingBytesBeforeCounters = | ||
|
|
@@ -142,34 +171,52 @@ void __llvm_profile_get_padding_sizes_for_counters( | |
| *PaddingBytesAfterBitmapBytes = | ||
| calculateBytesNeededToPageAlign(NumBitmapBytes); | ||
| *PaddingBytesAfterNames = calculateBytesNeededToPageAlign(NamesSize); | ||
| // Set these two variables to zero to avoid uninitialized variables | ||
| // even if VTableSize and VNameSize are known to be zero. | ||
| if (PaddingBytesAfterVTable != NULL) | ||
| *PaddingBytesAfterVTable = 0; | ||
| if (PaddingBytesAfterVName != NULL) | ||
| *PaddingBytesAfterVName = 0; | ||
| return 0; | ||
| } | ||
|
|
||
| COMPILER_RT_VISIBILITY | ||
| uint64_t __llvm_profile_get_size_for_buffer_internal( | ||
| const __llvm_profile_data *DataBegin, const __llvm_profile_data *DataEnd, | ||
| const char *CountersBegin, const char *CountersEnd, const char *BitmapBegin, | ||
| const char *BitmapEnd, const char *NamesBegin, const char *NamesEnd, | ||
| const VTableProfData *VTableBegin, const VTableProfData *VTableEnd, | ||
| const char *VNamesBegin, const char *VNamesEnd) { | ||
| /* Match logic in __llvm_profile_write_buffer(). */ | ||
| const uint64_t NamesSize = (NamesEnd - NamesBegin) * sizeof(char); | ||
| uint64_t DataSize = __llvm_profile_get_data_size(DataBegin, DataEnd); | ||
| uint64_t CountersSize = | ||
| __llvm_profile_get_counters_size(CountersBegin, CountersEnd); | ||
| const uint64_t NumBitmapBytes = | ||
| __llvm_profile_get_num_bitmap_bytes(BitmapBegin, BitmapEnd); | ||
| const uint64_t VTableSize = | ||
| __llvm_profile_get_vtable_section_size(VTableBegin, VTableEnd); | ||
| const uint64_t VNameSize = | ||
| __llvm_profile_get_name_size(VNamesBegin, VNamesEnd); | ||
|
|
||
| /* Determine how much padding is needed before/after the counters and after | ||
| * the names. */ | ||
| uint64_t PaddingBytesBeforeCounters, PaddingBytesAfterCounters, | ||
| PaddingBytesAfterNames, PaddingBytesAfterBitmapBytes, | ||
| PaddingBytesAfterVTable, PaddingBytesAfterVNames; | ||
| __llvm_profile_get_padding_sizes_for_counters( | ||
| DataSize, CountersSize, NumBitmapBytes, NamesSize, 0 /* VTableSize */, | ||
| 0 /* VNameSize */, &PaddingBytesBeforeCounters, | ||
| &PaddingBytesAfterCounters, &PaddingBytesAfterBitmapBytes, | ||
| &PaddingBytesAfterNames, &PaddingBytesAfterVTable, | ||
| &PaddingBytesAfterVNames); | ||
|
|
||
| return sizeof(__llvm_profile_header) + __llvm_write_binary_ids(NULL) + | ||
| DataSize + PaddingBytesBeforeCounters + CountersSize + | ||
| PaddingBytesAfterCounters + NumBitmapBytes + | ||
| PaddingBytesAfterBitmapBytes + NamesSize + PaddingBytesAfterNames + | ||
| VTableSize + PaddingBytesAfterVTable + VNameSize + | ||
| PaddingBytesAfterVNames; | ||
| } | ||
|
|
||
| COMPILER_RT_VISIBILITY | ||
|
|
@@ -191,7 +238,10 @@ COMPILER_RT_VISIBILITY int __llvm_profile_write_buffer_internal( | |
| const char *NamesBegin, const char *NamesEnd) { | ||
| ProfDataWriter BufferWriter; | ||
| initBufferWriter(&BufferWriter, Buffer); | ||
| // Set virtual table arguments to NULL since they are not supported yet. | ||
| return lprofWriteDataImpl( | ||
| &BufferWriter, DataBegin, DataEnd, CountersBegin, CountersEnd, | ||
| BitmapBegin, BitmapEnd, 0 /* VPDataReader */, NamesBegin, NamesEnd, | ||
mingmingl-llvm marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| NULL /* VTableBegin */, NULL /* VTableEnd */, NULL /* VNamesBegin */, | ||
| NULL /* VNamesEnd */, 0 /* SkipNameDataWrite */); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
indentation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed.