Clickhouse corrupt after server failure

[RoyvanEmpel]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

What are you trying to accomplish?

Hi,

The server that our self hosted sentry runt on had a hardware failure causing it to shutdown. After getting everything back up and running sentry no longer starts up.

Running docker compose up -d i get the following output:

dependency failed to start: container sentry-self-hosted-clickhouse-1 is unhealthy

I tried reinstalling using ./install.sh and after that i tried updating it to the latest version but i again get this output:

▶ Bootstrapping and migrating Snuba ...
Container sentry-self-hosted-zookeeper-1 Creating
Container sentry-self-hosted-redis-1 Creating
Container sentry-self-hosted-clickhouse-1 Creating
Container sentry-self-hosted-redis-1 Created
Container sentry-self-hosted-zookeeper-1 Created
Container sentry-self-hosted-kafka-1 Creating
Container sentry-self-hosted-clickhouse-1 Created
Container sentry-self-hosted-kafka-1 Created
Container sentry-self-hosted-clickhouse-1 Starting
Container sentry-self-hosted-zookeeper-1 Starting
Container sentry-self-hosted-redis-1 Starting
Container sentry-self-hosted-redis-1 Started
Container sentry-self-hosted-clickhouse-1 Started
Container sentry-self-hosted-zookeeper-1 Started
Container sentry-self-hosted-zookeeper-1 Waiting
Container sentry-self-hosted-zookeeper-1 Healthy
Container sentry-self-hosted-kafka-1 Starting
Container sentry-self-hosted-kafka-1 Started
dependency failed to start: container sentry-self-hosted-clickhouse-1 is unhealthy
Error in install/bootstrap-snuba.sh:3.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 1
-> ./install.sh:main:32
--> install/bootstrap-snuba.sh:source:3

Cleaning up...

How are you getting stuck?

Because of this sentry isn't starting. It seems that the clickhouse DB is corrupted, can i just drop the files of it and will it recover? I am guessing that issues are stored in it.
Maybe there is a command to try a self-repair?

Oh and BTW:
this says that the latest version is 24.3.0 while it is 24.2.0
https://develop.sentry.dev/self-hosted/releases/

Where in the product are you?

Other

Link

No response

DSN

No response

Version

24.2.0

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[RoyvanEmpel]

(version 21.8.13.1.altinitystable (altinity build))
2024.02.26 10:39:41.282861 [ 266 ] {} default.replays_local (38df915e-e158-46ca-b8df-915ee15806ca): Detaching broken part /var/lib/clickhouse/store/38d/38df915e-e158-46ca-b8df-915ee15806ca/90-20240226_268916_268916_0. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
2024.02.26 10:39:41.283485 [ 300 ] {} auto DB::MergeTreeData::loadDataParts(bool)::(anonymous class)::operator()() const: Code: 27, e.displayText() = DB::ParsingException: Cannot parse input: expected 'columns format version: 1\n' at end of stream., Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8fe7c9a in /usr/bin/clickhouse

1. DB::throwAtAssertionFailed(char const*, DB::ReadBuffer&) @ 0x9041917 in /usr/bin/clickhouse
2. DB::NamesAndTypesList::readText(DB::ReadBuffer&) @ 0xfcb6a18 in /usr/bin/clickhouse
3. DB::IMergeTreeDataPart::loadColumns(bool) @ 0x10d416db in /usr/bin/clickhouse
4. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x10d40b89 in /usr/bin/clickhouse
5. ? @ 0x10dd672f in /usr/bin/clickhouse
6. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x902b638 in /usr/bin/clickhouse
7. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()&&...)::'lambda'()::operator()() @ 0x902d1df in /usr/bin/clickhouse
8. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x902891f in /usr/bin/clickhouse
9. ? @ 0x902c203 in /usr/bin/clickhouse
10. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
11. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    (version 21.8.13.1.altinitystable (altinity build))
    2024.02.26 10:39:41.283514 [ 300 ] {} default.replays_local (38df915e-e158-46ca-b8df-915ee15806ca): Detaching broken part /var/lib/clickhouse/store/38d/38df915e-e158-46ca-b8df-915ee15806ca/90-20240226_265746_268912_786. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
    2024.02.26 10:39:41.283524 [ 294 ] {} auto DB::MergeTreeData::loadDataParts(bool)::(anonymous class)::operator()() const: Code: 27, e.displayText() = DB::ParsingException: Cannot parse input: expected 'columns format version: 1\n' at end of stream., Stack trace (when copying this message, always include the lines below):
12. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8fe7c9a in /usr/bin/clickhouse
13. DB::throwAtAssertionFailed(char const*, DB::ReadBuffer&) @ 0x9041917 in /usr/bin/clickhouse
14. DB::NamesAndTypesList::readText(DB::ReadBuffer&) @ 0xfcb6a18 in /usr/bin/clickhouse
15. DB::IMergeTreeDataPart::loadColumns(bool) @ 0x10d416db in /usr/bin/clickhouse
16. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x10d40b89 in /usr/bin/clickhouse
17. ? @ 0x10dd672f in /usr/bin/clickhouse
18. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x902b638 in /usr/bin/clickhouse
19. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()&&...)::'lambda'()::operator()() @ 0x902d1df in /usr/bin/clickhouse
20. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x902891f in /usr/bin/clickhouse
21. ? @ 0x902c203 in /usr/bin/clickhouse
22. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
23. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    (version 21.8.13.1.altinitystable (altinity build))
    2024.02.26 10:39:41.283550 [ 294 ] {} default.replays_local (38df915e-e158-46ca-b8df-915ee15806ca): Detaching broken part /var/lib/clickhouse/store/38d/38df915e-e158-46ca-b8df-915ee15806ca/90-20240226_268908_268908_0. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
    2024.02.26 10:39:41.285074 [ 248 ] {} auto DB::MergeTreeData::loadDataParts(bool)::(anonymous class)::operator()() const: Code: 27, e.displayText() = DB::ParsingException: Cannot parse input: expected 'columns format version: 1\n' at end of stream., Stack trace (when copying this message, always include the lines below):
24. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8fe7c9a in /usr/bin/clickhouse
25. DB::throwAtAssertionFailed(char const*, DB::ReadBuffer&) @ 0x9041917 in /usr/bin/clickhouse
26. DB::NamesAndTypesList::readText(DB::ReadBuffer&) @ 0xfcb6a18 in /usr/bin/clickhouse
27. DB::IMergeTreeDataPart::loadColumns(bool) @ 0x10d416db in /usr/bin/clickhouse
28. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x10d40b89 in /usr/bin/clickhouse
29. ? @ 0x10dd672f in /usr/bin/clickhouse
30. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x902b638 in /usr/bin/clickhouse
31. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()&&...)::'lambda'()::operator()() @ 0x902d1df in /usr/bin/clickhouse
32. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x902891f in /usr/bin/clickhouse
33. ? @ 0x902c203 in /usr/bin/clickhouse
34. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
35. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    (version 21.8.13.1.altinitystable (altinity build))
    2024.02.26 10:39:41.285096 [ 248 ] {} default.replays_local (38df915e-e158-46ca-b8df-915ee15806ca): Detaching broken part /var/lib/clickhouse/store/38d/38df915e-e158-46ca-b8df-915ee15806ca/90-20240226_268913_268913_0. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
    2024.02.26 10:39:41.285278 [ 248 ] {} auto DB::MergeTreeData::loadDataParts(bool)::(anonymous class)::operator()() const: Code: 27, e.displayText() = DB::ParsingException: Cannot parse input: expected 'columns format version: 1\n' at end of stream., Stack trace (when copying this message, always include the lines below):
36. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8fe7c9a in /usr/bin/clickhouse
37. DB::throwAtAssertionFailed(char const*, DB::ReadBuffer&) @ 0x9041917 in /usr/bin/clickhouse
38. DB::NamesAndTypesList::readText(DB::ReadBuffer&) @ 0xfcb6a18 in /usr/bin/clickhouse
39. DB::IMergeTreeDataPart::loadColumns(bool) @ 0x10d416db in /usr/bin/clickhouse
40. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x10d40b89 in /usr/bin/clickhouse
41. ? @ 0x10dd672f in /usr/bin/clickhouse
42. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x902b638 in /usr/bin/clickhouse
43. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()&&...)::'lambda'()::operator()() @ 0x902d1df in /usr/bin/clickhouse
44. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x902891f in /usr/bin/clickhouse
45. ? @ 0x902c203 in /usr/bin/clickhouse
46. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
47. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    (version 21.8.13.1.altinitystable (altinity build))
    2024.02.26 10:39:41.285303 [ 248 ] {} default.replays_local (38df915e-e158-46ca-b8df-915ee15806ca): Detaching broken part /var/lib/clickhouse/store/38d/38df915e-e158-46ca-b8df-915ee15806ca/90-20240226_268905_268905_0. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
    2024.02.26 10:39:41.294025 [ 297 ] {} auto DB::MergeTreeData::loadDataParts(bool)::(anonymous class)::operator()() const: Code: 27, e.displayText() = DB::ParsingException: Cannot parse input: expected 'columns format version: 1\n' at end of stream., Stack trace (when copying this message, always include the lines below):
48. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8fe7c9a in /usr/bin/clickhouse
49. DB::throwAtAssertionFailed(char const*, DB::ReadBuffer&) @ 0x9041917 in /usr/bin/clickhouse
50. DB::NamesAndTypesList::readText(DB::ReadBuffer&) @ 0xfcb6a18 in /usr/bin/clickhouse
51. DB::IMergeTreeDataPart::loadColumns(bool) @ 0x10d416db in /usr/bin/clickhouse
52. DB::IMergeTreeDataPart::loadColumnsChecksumsIndexes(bool, bool) @ 0x10d40b89 in /usr/bin/clickhouse
53. ? @ 0x10dd672f in /usr/bin/clickhouse
54. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x902b638 in /usr/bin/clickhouse
55. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()&&...)::'lambda'()::operator()() @ 0x902d1df in /usr/bin/clickhouse
56. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x902891f in /usr/bin/clickhouse
57. ? @ 0x902c203 in /usr/bin/clickhouse
58. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
59. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    (version 21.8.13.1.altinitystable (altinity build))
    2024.02.26 10:39:41.294052 [ 297 ] {} default.replays_local (38df915e-e158-46ca-b8df-915ee15806ca): Detaching broken part /var/lib/clickhouse/store/38d/38df915e-e158-46ca-b8df-915ee15806ca/90-20240226_265746_268907_781. If it happened after update, it is likely because of backward incompability. You need to resolve this manually
    2024.02.26 10:39:41.318674 [ 44 ] {} Application: Caught exception while loading metadata: Code: 231, e.displayText() = DB::Exception: Suspiciously many (30) broken parts to remove.: Cannot attach table default.replays_local from metadata file /var/lib/clickhouse/store/d48/d48a4d38-4b97-43bb-948a-4d384b9733bb/replays_local.sql from query ATTACH TABLE default.replays_local UUID '38df915e-e158-46ca-b8df-915ee15806ca' (replay_id UUID, debug_id UUID, count_info_events UInt8 MATERIALIZED (debug_id != '00000000-0000-0000-0000-000000000000') + (info_id != '00000000-0000-0000-0000-000000000000'), count_warning_events UInt8 MATERIALIZED warning_id != '00000000-0000-0000-0000-000000000000', count_error_events UInt8 MATERIALIZED (error_id != '00000000-0000-0000-0000-000000000000') + (fatal_id != '00000000-0000-0000-0000-000000000000'), info_id UUID, warning_id UUID, error_id UUID, fatal_id UUID, replay_type LowCardinality(Nullable(String)), error_sample_rate Nullable(Float64), session_sample_rate Nullable(Float64), event_hash UUID, segment_id Nullable(UInt16), trace_ids Array(UUID), _trace_ids_hashed Array(UInt64) MATERIALIZED arrayMap(t -> cityHash64(t), trace_ids), title Nullable(String), url Nullable(String), urls Array(String), is_archived Nullable(UInt8), error_ids Array(UUID), _error_ids_hashed Array(UInt64) MATERIALIZED arrayMap(t -> cityHash64(t), error_ids), project_id UInt64, timestamp DateTime, replay_start_timestamp Nullable(DateTime), platform LowCardinality(String), environment LowCardinality(Nullable(String)), release Nullable(String), dist Nullable(String), ip_address_v4 Nullable(IPv4), ip_address_v6 Nullable(IPv6), user Nullable(String), user_id Nullable(String), user_name Nullable(String), user_email Nullable(String), os_name LowCardinality(Nullable(String)), os_version Nullable(String), browser_name LowCardinality(Nullable(String)), browser_version Nullable(String), device_name LowCardinality(Nullable(String)), device_brand LowCardinality(Nullable(String)), device_family LowCardinality(Nullable(String)), device_model LowCardinality(Nullable(String)), sdk_name LowCardinality(Nullable(String)), sdk_version LowCardinality(Nullable(String)), tags.key Array(String), tags.value Array(String), click_node_id UInt32 DEFAULT 0, click_tag LowCardinality(String) DEFAULT '', click_id String DEFAULT '', click_class Array(String), click_text String DEFAULT '', click_role LowCardinality(String) DEFAULT '', click_alt String DEFAULT '', click_testid String DEFAULT '', click_aria_label String DEFAULT '', click_title String DEFAULT '', click_component_name String, click_is_dead UInt8, click_is_rage UInt8, count_errors UInt16 MATERIALIZED length(error_ids), count_urls UInt16 MATERIALIZED length(urls), retention_days UInt16, partition UInt16, offset UInt64, INDEX bf_trace_ids_hashed _trace_ids_hashed TYPE bloom_filter GRANULARITY 1, INDEX bf_error_ids_hashed _error_ids_hashed TYPE bloom_filter GRANULARITY 1, INDEX bf_user_id user_id TYPE bloom_filter GRANULARITY 1, INDEX bf_user_email user_email TYPE bloom_filter GRANULARITY 1, INDEX bf_ip_address_v4 ip_address_v4 TYPE bloom_filter GRANULARITY 1) ENGINE = ReplacingMergeTree PARTITION BY (retention_days, toMonday(timestamp)) ORDER BY (project_id, toStartOfDay(timestamp), cityHash64(replay_id), event_hash) TTL timestamp + toIntervalDay(retention_days) SETTINGS index_granularity = 8192: while loading database default from path /var/lib/clickhouse/metadata/default, Stack trace (when copying this message, always include the lines below):
60. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8fe7c9a in /usr/bin/clickhouse
61. DB::MergeTreeData::loadDataParts(bool) @ 0x10d88537 in /usr/bin/clickhouse
62. DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, DB::StorageInMemoryMetadata const&, bool, std::__1::shared_ptrDB::Context, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, DB::MergeTreeData::MergingParams const&, std::__1::unique_ptr<DB::MergeTreeSettings, std::__1::default_deleteDB::MergeTreeSettings >, bool) @ 0x10fb2857 in /usr/bin/clickhouse
63. ? @ 0x10fa7ce7 in /usr/bin/clickhouse
64. DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptrDB::Context, std::__1::shared_ptrDB::Context, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const @ 0x10a44ec1 in /usr/bin/clickhouse
65. DB::createTableFromAST(DB::ASTCreateQuery, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptrDB::Context, bool) @ 0xff9c705 in /usr/bin/clickhouse
66. ? @ 0xff9a853 in /usr/bin/clickhouse
67. ? @ 0xff9b83f in /usr/bin/clickhouse
68. ThreadPoolImpl::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x902b638 in /usr/bin/clickhouse
69. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()>(void&&, void ThreadPoolImpl::scheduleImpl(std::__1::function<void ()>, int, std::__1::optional)::'lambda0'()&&...)::'lambda'()::operator()() @ 0x902d1df in /usr/bin/clickhouse
70. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x902891f in /usr/bin/clickhouse
71. ? @ 0x902c203 in /usr/bin/clickhouse
72. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
73. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    (version 21.8.13.1.altinitystable (altinity build))
    ������
    �2024.02.26 10:39:41.320792 [ 44 ] {} Application: DB::Exception: Suspiciously many (30) broken parts to remove.: Cannot attach table default.replays_local from metadata file /var/lib/clickhouse/store/d48/d48a4d38-4b97-43bb-948a-4d384b9733bb/replays_local.sql from query ATTACH TABLE default.replays_local UUID '38df915e-e158-46ca-b8df-915ee15806ca' (replay_id UUID, debug_id UUID, count_info_events UInt8 MATERIALIZED (debug_id != '00000000-0000-0000-0000-000000000000') + (info_id != '00000000-0000-0000-0000-000000000000'), count_warning_events UInt8 MATERIALIZED warning_id != '00000000-0000-0000-0000-000000000000', count_error_events UInt8 MATERIALIZED (error_id != '00000000-0000-0000-0000-000000000000') + (fatal_id != '00000000-0000-0000-0000-000000000000'), info_id UUID, warning_id UUID, error_id UUID, fatal_id UUID, replay_type LowCardinality(Nullable(String)), error_sample_rate Nullable(Float64), session_sample_rate Nullable(Float64), event_hash UUID, segment_id Nullable(UInt16), trace_ids Array(UUID), _trace_ids_hashed Array(UInt64) MATERIALIZED arrayMap(t -> cityHash64(t), trace_ids), title Nullable(String), url Nullable(String), urls Array(String), is_archived Nullable(UInt8), error_ids Array(UUID), _error_ids_hashed Array(UInt64) MATERIALIZED arrayMap(t -> cityHash64(t), error_ids), project_id UInt64, timestamp DateTime, replay_start_timestamp Nullable(DateTime), platform LowCardinality(String), environment LowCardinality(Nullable(String)), release Nullable(String), dist Nullable(String), ip_address_v4 Nullable(IPv4), ip_address_v6 Nullable(IPv6), user Nullable(String), user_id Nullable(String), user_name Nullable(String), user_email Nullable(String), os_name LowCardinality(Nullable(String)), os_version Nullable(String), browser_name LowCardinality(Nullable(String)), browser_version Nullable(String), device_name LowCardinality(Nullable(String)), device_brand LowCardinality(Nullable(String)), device_family LowCardinality(Nullable(String)), device_model LowCardinality(Nullable(String)), sdk_name LowCardinality(Nullable(String)), sdk_version LowCardinality(Nullable(String)), tags.key Array(String), tags.value Array(String), click_node_id UInt32 DEFAULT 0, click_tag LowCardinality(String) DEFAULT '', click_id String DEFAULT '', click_class Array(String), click_text String DEFAULT '', click_role LowCardinality(String) DEFAULT '', click_alt String DEFAULT '', click_testid String DEFAULT '', click_aria_label String DEFAULT '', click_title String DEFAULT '', click_component_name String, click_is_dead UInt8, click_is_rage UInt8, count_errors UInt16 MATERIALIZED length(error_ids), count_urls UInt16 MATERIALIZED length(urls), retention_days UInt16, partition UInt16, offset UInt64, INDEX bf_trace_ids_hashed _trace_ids_hashed TYPE bloom_filter GRANULARITY 1, INDEX bf_error_ids_hashed _error_ids_hashed TYPE bloom_filter GRANULARITY 1, INDEX bf_user_id user_id TYPE bloom_filter GRANULARITY 1, INDEX bf_user_email user_email TYPE bloom_filter GRANULARITY 1, INDEX bf_ip_address_v4 ip_address_v4 TYPE bloom_filter GRANULARITY 1) ENGINE = ReplacingMergeTree PARTITION BY (retention_days, toMonday(timestamp)) ORDER BY (project_id, toStartOfDay(timestamp), cityHash64(replay_id), event_hash) TTL timestamp + toIntervalDay(retention_days) SETTINGS index_granularity = 8192: while loading database default from path /var/lib/clickhouse/metadata/default

[getsantry]

Routing to @getsentry/product-owners-settings-integrations for triage ⏲️

[hubertdeng123]

You may need to manually remove rows from your clickhouse table since it looks like corrupt data got in. replays_local seems to be the table in particular.

[RoyvanEmpel]

Hey thanks for the response. Nobody in our company has worked with clickhouse before. So we don't know how the database functions or we can connect to it. Do we just drop files somewhere?

We don't really care about the existing replays and are wondering if there is a way to simply wipe the table so it recreates it. We just want the selfhosted sentry up and running.