Introduction

This is the first public release of CogCache and it’s considered to be MVP.

What is CogCache?

CogCache is a fully managed, state-of-the-art, in-memory, Enterprise-grade Azure OpenAI caching solution, which also provides for all the AI operations full predictability, control, transparency and remediation.

What are the CogCache benefits?

Caching

• Faster OpenAI Azure-based responses. - No more waiting time for streams of text to render, responses are generated in the blink of an eye.

• Cost reductions - Minimize costs and reduce carbon footprint by eliminating the need to consume Azure OpenAI tokens on repeated prompts.

Responsible AI

• Transparency - Empower compliance, legal and security teams with the ability to monitor generated content, ensuring transparency and control over AI processes.

• Remediation - CogCache permits human-approved adjustments for cache integrity
and system reliability.

• Full predictability - Always get the same response for the same prompt. No more stochastic responses.

Azure Marketplace

This release introduces official support for the Azure Marketplace for CogCache.

Things you should know:

• you can purchase and manage a CogCache subscription from the Azure Marketplace; this is a self-serve approach and requires no Touchcast intervention in the process

• there is an automatic onboarding flow for the subscription purchased through the Azure Marketplace which entails the following:

  • creation a CogCache organization and making the user who purchased the subscription as owner of the organization

  • creation of the first CogCache application with the corresponding Proxy API key

  • step by step guide for how to integrate CogCache with your Azure OpenAI based application

• the billing part (invoices, handling payments, etc..) is handled by Microsoft entirely.

CMS

This represents the area of CogCache where users can manage their CogCache account, their CogCache organization (branding, API keys, members), view the cache entries, etc…

The CMS comes with the following features:

Applications

An application is the way we group cache entries together, to create proper segregation of data. When Proxy API calls are made they are authenticated through API keys which are connected to applications.

Open

Cache

This is the place where you can view and manage all the cache entries.

Cache entries will appear automatically in here once the Proxy API is integrated with your application.

Additionally, you can also manually add cache entries, one at a time, or via CSV import.

Open

Listing of cache items with ability to search by prompts and filter by applications

Organization / Settings

This is the place that allows managing organization settings. For now, just the ability to change the name.

Organization / Keys

This section allows you create, update and delete API keys for both Proxy API and Management API.

Proxy API keys are used with the Proxy API and they are associated with an application.

Management API keys are used with the Management API and they are global for the entire organization, not restricted to a specific application.

Organization / Members

From this page you can manage members of the organizations, such as view list of existing members, adding new ones or even deleting them.

Organization / Billing & usage

On this page you can view the current plan you’re using and cache items usage information.

My account

This is the place where you can manage your CogCache account.

Proxy API

The CogCache Proxy API call, once integrated, helps developers to speed up Azure OpenAI calls and reduce costs at the same time.

This API has identical specifications as the Azure OpenAI API and works as the man-in-the-middle (a.k.a proxy) between applications and any Azure OpenAI deployment.

When a call is made to the Azure OpenAI through CogCache Proxy API, the system first checks to see if there is a response in the cache for the given prompt, and if so, return it, otherwise it will reach out to the Azure OpenAI deployment to fetch the response, cache it, and return it back to the application.

Management API

The Management API is meant to be used to manage content for your CogCache organization. It has the following capabilities, with the note that we will expand this in the future:

1. list, update, create, delete cache items

2. list, update, create, delete applications

Documentation

Public developers documentation

Can be found here and is meant to help developers integrate CogCache with their applications.

Others

Limitations

This release has the following limitations:

• CogCache currently works with single queries. It doesn’t work with a conversation and with RAG use cases. Support for those use cases will be added in a future release.

• CogCache similarity threshold is currently hardcoded at 82%. This applies when we try to search for a cache entry using semantic search. We plan to expose this parameter as an application setting.