Implement Initial Search Data Store

[FlatBallFlyer]

Implement the database necessary to support a search feature. The search data store should consume state change events from the event bus to maintain a copy of all relevant objects. Objects should be indexed as a single polymorphic list that supports a simple text search for a value anywhere in the store. Support for wild-card or RegEx searches is desirable but not required.

Related Learning Resources

• Review mentorhub-mongodb repository
• Getting Started with Docker
• Run OpenSearch in a Container
• Getting Started with OpenSearch
• What is Amazon EventBridge
• What is Amazon Kinesis
• Getting Started with Lamda

Tasks

• Finalize Architecture DocumentDB -> EventBridge -> Lamda -> OpenSearch
• Update Architecture Diagrams with findings
• Create Containerized Database, with test data loaded.
• Update remaining Search Issues and move to On-Deck
• Work with SRE to provision Search for Dev Environment
• Create event streams, and Lamda transformations
• Test and Verify data streams, and transformations

[FlatBallFlyer]

Capturing clarifications from a chat: This repo should have CI that builds and publishes a container that connects to an openSearch instance based on environment variables, and then defines indexes, creates maps, and makes other configurations needed to initialize the solution, and then optionally load test data. The scripts need to make sure they need to run before making any configuration changes (versioning?). You will also need to update the system docker compose file in the mentorHub repo to start your containers with a new tag - search-db?

See the mongdb repo migrate.js file for an example of code that performs similar work there. That is mongosh code, you will use some form of openSearch API

[MyNameIsAndrew-Mangix]

I thought I got it working, but it was VERY bad. I thought I was loading data but I was creating a mapping for every document. I'm trying to ingest the test data from the mongo container as a baseline for the mapping/indexing functionality since the shapes of the data I need to take right now are there.

I have a few solutions in mind for creating the mapping, but I'm not sure what would be best. Right now I'm trying to see if I can create a set of unique data shapes to minimize the mappings created, since some documents use more of the schema than others, but it feels bad because I'm probably going to iterate over every document. I thinking about if using the schema json for mapping is a possibility, but I don't know how I'd get them. I could use a quick cUrl or something, but I don't know. Everything I think of feels a bit icky. At the moment I'm taking a second look at dynamic mapping. Originally I thought it wouldn't work for the use I'm hoping for, but it might.

[FlatBallFlyer]

Ok, so we can do a better job of documenting this standard, but every collection will have an _id and name attribute. The ingest should add a collection attribute to indicate which collection the data came from. Following the KISS principles, since our baseline search feature will be searching only Names we can create a map for just name and collection attributes and get something working. We can increase the complexity of search and mapping in future iterations.