MATCHA Chart Summarizer: Automatic Summaries for Data Visualizations

About

MATCHA Chart Summarizer is a Chrome Extension that helps blind, low-vision, and neurodivergent screen reader users access the meaning of charts and graphs on websites without needing to see the visuals. Its functionality includes the following:
- Allows the user to keyboard-focus on images, including data visualizations.
- Provides a command (Alt+O, or Command+Shift+O on macOS) to generate an informative, spoken summary of a chart image using an LLM (Google Gemini).
- Provides a command (Alt+L, or Command+Shift+L on macOS) for the user to ask natural language questions about the chart via voice input.
- Converts all outputs to audible speech in real-time for screen reader accessibility.

In more technical terms, the `read_content.js` content script:
- Makes images keyboard-focusable via `tabindex=0`.
- When a user focuses on an image and requests a summary, the image is converted to Base64 and passed to Gemini with a descriptive prompt. If the image is a chart, the resulting summary is read aloud using Chrome TTS.
- When a question is asked, voice input is transcribed using the SpeechRecognition interface. Gemini is prompted to answer the question based on the chart image. The response is spoken back to the user using Chrome TTS.

MATCHA Chart Summarizer has been tested to work on a multitude of popular websites, including but not limited to Gmail.com, Reddit.com, Instagram.com, Weather.com, and Wikipedia.com.

Overall, this extension gives everyone equal access to the data behind the images, making the web more inclusive, one chart at a time.

MATCHA Chart Summarizer is open source. Check out the GitHub repository for our project here: https://github.com/om-arya/matcha