Perplexity Sonar API Hackathon Submission - Koushik Kr. Bug

Project Overview

This repository contains the source code modifications and additions I made to the Chromium browser as part of the Perplexity Sonar API Hackathon. The goal was to natively integrate Perplexity's Sonar API to enhance browsing capabilities.

"Important Note on Repository Structure": Due to very large size of the full Chromium source code (approx. 155GB) and GitHub's repository size limitations, this repository does not contain the entire Chromium source tree. Instead, it includes only the specific files I created or modified for the hackathon features. I could have uploaded my whole patch, to google gerrit, but that would amke the source code public, which would have defeated the purpose and rules of the hackathon. The directory structure within this repository mirrors the structure found within a standard chromium/src checkout.

This detailed README provides a walkthrough of the implemented features, the key code changes, and how one can conceptually integrate these changes into a full Chromium checkout.

Features Implemented

Demonstration Video

1. Sonar-Powered Omnibox Answers: Users can type sonar followed by a query in the Omnibox (address bar) to get direct answers from the Sonar API displayed as an autocomplete suggestion.
2. Context Menu: "Summarize with Sonar": Users can select text on any webpage, right-click, and choose "Summarize with Sonar." The selected text is sent to the Sonar API, and the summary is displayed in a new browser tab.

Prerequisites for Replication/Understanding

To build and run these features within Chromium, you would generally need:

1. A Chromium Development Environment: I have built the chromium using wsl2 linux ubuntu 22.04 Follow the official Chromium instructions to check out and build the browser:
   • Get the Code and Build: Checking out and building Chromium on Linux
2. Perplexity Sonar API Key: You will need a valid API key from Perplexity AI.
   • Check official docs: SONAR by Perplexity

Directory Structure of This Repository

All paths mentioned below are relative to the chromium/src/ (very important) directory of a standard Chromium checkout. The files in this repository should be placed into their corresponding locations within your own Chromium checkout. I am using chromium Version 138.0.7160.0 (Developer Build) (64-bit)

Key directories you'll find in this repository containing my changes:

• components/omnibox/browser/
• chrome/browser/renderer_context_menu/
• chrome/app/
• chrome/browser/ui/actions
• tools/metrics/histograms/metadata/ui/

Feature 1: Sonar-Powered Omnibox Answers

Goal

To provide instant answers directly in the Omnibox (address bar) suggestions when a user types a query prefixed with the sonar keyword.

Key Files Involved (from this repository)

• components/omnibox/browser/sonar_autocomplete_provider.h
• components/omnibox/browser/sonar_autocomplete_provider.cc
• components/omnibox/browser/autocomplete_controller.cc
• components/omnibox/browser/BUILD.gn

Code Explanation & Integration Steps

1. components/omnibox/browser/sonar_autocomplete_provider.h:

   • Declares the SonarAutocompleteProvider class, inheriting from AutocompleteProvider.
   • Member variables:
     • client_ and listener_: Standard for providers.
     • loader_: A std::unique_ptr<network::SimpleURLLoader> for the API call.
     • debounce_timer_: A base::OneShotTimer to delay API calls.
     • current_input_, current_query_: Stores input between debounces.
     • weak_ptr_factory_: For safe asynchronous callbacks.

2. components/omnibox/browser/sonar_autocomplete_provider.cc:

   • Constants:
     • kSonarKeyword (u"sonar"): The trigger keyword.
     • kSonarDebounceDelay (base::Milliseconds(300)): Delay before sending API request.
     • kMinQueryLength = 3: Minimum query length after the keyword.
   • Constructor SonarAutocompleteProvider(...):
     • Initializes the base class with Type::TYPE_SEARCH.
     • Stores client and listener.
   • Start(const AutocompleteInput& input, bool minimal_changes) Method:
     • Logs input and checks input.omit_asynchronous_matches().
     • Uses AutocompleteInput::ExtractKeywordFromInput() to check for kSonarKeyword and extract the remaining_input_text.
     • If not a valid sonar query, clears matches and sets done_ = true.
     • If valid, sets done_ = false, stops and restarts debounce_timer_ if remaining_input_text.length() >= kMinQueryLength.
     • Stores input and processed remaining_input_text (whitespace trimmed, trailing '?' removed) in current_input_ and current_query_.
   • ExecuteQuery() Method (called by debounce_timer_):
     • Resets loader_ and invalidates weak pointers.
     • Defines net::NetworkTrafficAnnotationTag.
     • Creates network::ResourceRequest for https://api.perplexity.ai/chat/completions (POST).
     • Sets Authorization header with my API key and Content-Type.
     • Constructs the JSON request body:

       {
         "model": "sonar-pro",
         "messages": [
           { "role": "system", "content": "Be precise and concise." },
           { "role": "user", "content": "<user's query>" }
         ]
       }

     • Creates network::SimpleURLLoader, attaches upload string, sets timeout, and retry options.
     • Calls loader_->DownloadToString() binding OnApiReply with original_input_text (full "sonar query") and query_text (only just the part after "sonar").
   • OnApiReply(const std::u16string& original_input_text, ...) Method:
     • Sets done_ = true.
     • Performs error checking on the network response (loader_->NetError(), HTTP status code).
     • Parses the JSON response: base::JSONReader::Read(*response_body).
     • Extracts the answer text from answer_text = *content.
     • Cleans the answer text: base::CollapseWhitespace, AutocompleteMatch::SanitizeString, and custom removal of markdown like ** and citations [...].
     • Creates an AutocompleteMatch:
       • relevance = 1460 (high).
       • type = AutocompleteMatchType::HISTORY_EMBEDDINGS_ANSWER.
       • contents = cleaned Sonar answer.
       • fill_into_edit = original_input_text (e.g., "sonar what is X").
       • destination_url = Search URL for query_text (e.g., for "what is X").
       • description = u"Sonar Answer".
       • keyword = kSonarKeyword.
       • from_keyword = true.
       • transition = ui::PAGE_TRANSITION_KEYWORD.
     • Adds the match to matches_.
     • Calls listener_->OnProviderUpdate(true, this);.
   • Stop(AutocompleteStopReason stop_reason) Method:
     • Invalidates weak pointers, resets loader_, stops debounce_timer_, and calls base Stop().

3. Integration with components/omnibox/browser/autocomplete_controller.cc:

   • The SonarAutocompleteProvider is instantiated and added to the providers_ list, typically within the AutocompleteController::InitializeAsyncProviders() method, ensuring it participates in omnibox suggestions:

     // In AutocompleteController::InitializeAsyncProviders(...)
     if (provider_types & AutocompleteProvider::TYPE_SEARCH) { // Or a similar condition
       // ... other providers ...
       providers_.push_back(
           new SonarAutocompleteProvider(provider_client_.get(), this));
     }

4. Build System (components/omnibox/browser/BUILD.gn):

   • sonar_autocomplete_provider.cc and sonar_autocomplete_provider.h are added to the sources list of the appropriate source set (see static_library("browser")).

How to Replicate/Understand Sonar Omnibox Feature:

1. Create Files: Place sonar_autocomplete_provider.h and sonar_autocomplete_provider.cc in components/omnibox/browser/.
2. Modify BUILD.gn: Add the new source files and any missing dependencies to components/omnibox/browser/BUILD.gn.
3. Modify autocomplete_controller.cc: Include the sonar_autocomplete_provider.h header and add the instantiation line as shown above in InitializeAsyncProviders().
4. API Key: Replace your API key in sonar_autocomplete_provider.cc with a valid Perplexity API key.
5. Build Chromium.
6. Test: Type "sonar [your query]" in the omnibox.

---

Feature 2: Context Menu "Summarize with Sonar" (New Tab data: URL)

Goal

Allow users to select large & lengthy texts on a webpage, right-click, and select "Summarize with Sonar." The API-generated summary, along with the original text, is displayed in a new browser tab using a data: URL.

Key Files Involved (from this repository)

• chrome/browser/renderer_context_menu/render_view_context_menu.h
• chrome/browser/renderer_context_menu/render_view_context_menu.cc
• chrome/app/chrome_command_ids.h
• chrome/app/generated_resources.grd
• tools/metrics/histograms/metadata/ui/enums.xml

Code Explanation & Integration Steps

1. Define Command ID (chrome/app/chrome_command_ids.h):

   • A new command ID, IDC_CONTENT_CONTEXT_SUMMARIZE_WITH_SONAR, is defined at LINE 413 with

     #define IDC_CONTENT_CONTEXT_SUMMARIZE_WITH_SONAR 50180

2. Define Menu Item String (chrome/app/generated_resources.grd):

   • A string like IDS_CONTENT_CONTEXT_SUMMARIZE_WITH_SONAR with the value "Summarize with Sonar" has been added.

3. Add Members to RenderViewContextMenu Class (render_view_context_menu.h):

   • Declaration of the API callback method:

     void OnSonarApiResponseForSummarization(
         const std::string& original_text_utf8,
         std::unique_ptr<std::string> response_body);

   • Member variables for the API call:

     std::unique_ptr<network::SimpleURLLoader> sonnar_api_loader_;
     base::WeakPtrFactory<RenderViewContextMenu> summarize_weak_ptr_factory_{this};

4. Modify RenderViewContextMenu::InitMenu() (render_view_context_menu.cc):

   • The item is added conditionally when text is selected, often within the ITEM_GROUP_SEARCH_PROVIDER block:

     // Add "Summarize with Sonar" item if text is selected
     if (!params_.selection_text.empty()) {
     menu_model_.AddSeparator(ui::NORMAL_SEPARATOR);
     menu_model_.AddItem(IDC_CONTENT_CONTEXT_SUMMARIZE_WITH_SONAR,
                         u"Summarize with Sonar");
     }

5. Modify RenderViewContextMenu::IsCommandIdEnabled() (render_view_context_menu.cc):

   • Ensures the item is only enabled if text is selected:

     case IDC_CONTENT_CONTEXT_SUMMARIZE_WITH_SONAR:
       return !params_.selection_text.empty();

6. Modify RenderViewContextMenu::ExecuteCommand() (render_view_context_menu.cc):

   • A case for IDC_CONTENT_CONTEXT_SUMMARIZE_WITH_SONAR calls the new SummarizeWithSonar() method:

     case IDC_CONTENT_CONTEXT_SUMMARIZE_WITH_SONAR:
       SummarizeWithSonar();
       break;

7. Implement RenderViewContextMenu::SummarizeWithSonar() (render_view_context_menu.cc):

   • Retrieves params_.selection_text.
   • Resets any previous sonnar_api_loader_ and invalidates summarize_weak_ptr_factory_.
   • Sets up net::NetworkTrafficAnnotationTag.
   • Creates a network::ResourceRequest for https://api.perplexity.ai/chat/completions (POST).
   • Sets Authorization header with API key and Content-Type.
   • Constructs the JSON request body with a system prompt like "Please summarize the following text concisely." and the selected user text.
   • Creates network::SimpleURLLoader, attaches the upload string, sets timeout.
   • Calls loader_->DownloadToString(), binding RenderViewContextMenu::OnSonarApiResponseForSummarization and passing the selected_text_utf8 to it for later display.

8. Implement RenderViewContextMenu::OnSonarApiResponseForSummarization(...) (render_view_context_menu.cc):

   • This is the callback executed after the Sonar API call completes.
   • It checks for network or HTTP errors.
   • Parses the JSON response from Sonar to extract the summary from response.choices[0].message.content.
   • If parsing fails or API call errors occur, it sets summary_text_utf8 to an error message.
   • Dynamically constructs an HTML string for a data: URL. This HTML includes:
     • A title "Sonar Summary".
     • Basic CSS styling for cards and text.
     • A section for "Source Text" where original_text_utf8 is displayed.
     • A section for "Summary" where summary_text_utf8 (the actual Sonar summary or error) is displayed.
     • JavaScript to set the text content of these elements.
   • URL-encodes the generated HTML string.
   • Creates a data:text/html;charset=utf8,... URL.
   • The code opens this data_url in a NEW_FOREGROUND_TAB using browser->OpenURL().

9. UMA Logging (tools/metrics/histograms/metadata/ui/enums.xml):

   • An entry for IDC_CONTENT_CONTEXT_SUMMARIZE_WITH_SONAR (with value 156) is added to the RenderViewContextMenuItem enum to allow tracking its usage.
   • An entry for Summarize with Sonar (with value 31) is added to the ContextMenuOptionDesktop enum. SEE : <enum name="ContextMenuOptionDesktop">

How to Replicate/Understand Context Menu Summarization:

1. Modify chrome_command_ids.h: Define IDC_CONTENT_CONTEXT_SUMMARIZE_WITH_SONAR.
2. Modify String Resources (.grd file): Add IDS_CONTENT_CONTEXT_SUMMARIZE_WITH_SONAR.
3. Modify render_view_context_menu.h: Add the OnSonarApiResponseForSummarization declaration and the sonnar_api_loader_, summarize_weak_ptr_factory_ members.
4. Modify render_view_context_menu.cc:
   • Include necessary headers (base/json/json_reader.h, network headers, etc.).
   • Implement the changes in InitMenu(), IsCommandIdEnabled(), ExecuteCommand().
   • Implement the new SummarizeWithSonar() and OnSonarApiResponseForSummarization() methods as detailed above.
5. API Key: Insert a valid Perplexity API key in SummarizeWithSonar().
6. Modify enums.xml: Add the UMA entries.
7. Build Chromium.
8. Test: Select text on a webpage, right-click, and choose "Summarize with Sonar."

---

Building and Running

• After making the code changes as described, follow the standard Chromium build instructions for your platform.
• Crucially, replace placeholder API keys in both sonar_autocomplete_provider.cc and render_view_context_menu.cc with your valid Perplexity Sonar API key before building.

Notes and Limitations

• API Keys: API keys are currently hardcoded for this hackathon submission. In a production environment, these would be managed securely.
• Error Handling: Error handling for API calls is basic (logs errors, may show a generic error in the summary tab). More user-friendly UI for errors could be implemented.
• Context Menu Summary Display: The current context menu summary feature displays the result in a new data: URL tab. This is a straightforward approach for the hackathon. A more integrated solution might involve creating a dedicated WebUI called chrome://sonar-summary/ which is easy.
• Omnibox Match Type: The omnibox feature uses AutocompleteMatchType::HISTORY_EMBEDDINGS_ANSWER for display, which might have specific rendering characteristics. This could be changed to a more generic or custom answer type if needed.

Potential Future Work

• Integrate the context menu summary feature with a native Chromium side panel using the WebUI files (chrome/browser/ui/webui/side_panel/sonar_summary/ etc.)
• Provide user-facing settings for these features (e.g., to enable/disable, manage API keys securely).
• More sophisticated UI for displaying loading states and errors.

Acknowledgements

Thank you to the Perplexity AI team for organizing this hackathon and providing access to the Sonar API.