API Data Proxy

TLDR section at the bottom

Introduction: Why API Data Matters

In today's interconnected world, applications and organizations rely heavily on third-party APIs to drive decisions and operations. Whether it's integrating with external services or relying on vendor-provided applications, the data they provide forms a foundational layer for modern ecosystems.

Integrating external APIs into workflows presents challenges, especially in maintaining consistency, availability, and auditability of data. The API Data Proxy Pattern addresses these by capturing and internalizing third-party data, ensuring resilience and reliable data access.

The Business Case for Storing API Data

Imagine this scenario: you're using a third-party API to make critical business decisions, such as managing payment history information for a user from an external system. Months down the line, you may need to review those past decisions for an audit, compliance check, or strategic evaluation. Without access to the original data, understanding or justifying the choices made at that time becomes challenging.

Relying solely on an external system for critical data over time is risky. The solution is clear: you must bring the data in-house. Keeping a history of this data in-house safeguards against several potential pitfalls:

• Audits: With stored data, you can trace back to any point in time, preserving a clear record for audit purposes.
• Data Changes: Changes in API responses for the same request, even when valid, can complicate decision-making; with full history in-house, you can reconcile differences and make informed choices.
• Reporting and Analysis: Owning a copy of the data supports reporting and analytics beyond API limitations, such as REST constraints, by enabling independent queries.

In essence, storing API data internally ensures that data crucial to business decisions remains under your control. This isn't just an API issue - it's about data ownership and management.

Relying on third-party APIs can present other hidden challenges:

• Inconsistent Data: APIs don't always return consistent results, even for the same request. These inconsistencies can impact any critical outcomes based on this data.
• Undocumented Changes: APIs often evolve without notice, sometimes deviating from documentation, which can make data tracking difficult if changes aren't captured.

With these issues in mind, having the ability to track API data changes over time becomes essential. This is where an API Data Proxy adds value.

Introducing the API Data Proxy

The API Data Proxy provides a robust solution to these challenges by acting as a reverse proxy that captures and snapshots third-party API data into internal databases. This approach enables the following:

• Data Change Tracking: Track precisely when and how data has changed over time, making data mutability manageable.
• Data Availability: Access stored data independently of the API's availability or unexpected changes.
• Auditable Decisions: Maintain a complete historical record of the data used in business decisions at any given time.

The API Data Proxy ensures consistent, auditable access to critical external data, securing it for long-term use and analysis.

Technical Benefits: How the API Data Proxy Works

The core of the API Data Proxy is its ability to replicate data from the API into a database in a 1-to-1 like-for-like database schema. This means that the data inside your systems mirrors the entity and attribute structure of the API; there are no transformations (yet). Here's how this helps:

• Simplicity: The Proxy service simply transforms REST JSON into DB format - nothing else.
• Consistency: By keeping the structure identical, it becomes easier to maintain consistency between API responses and internal data representations.
• Future Proofing: When the API changes, you know exactly how your data and attributes evolve as the vendor improves or changes, making it easier to adjust.
• Offline Data Access: You can query historical data at any time without making additional API calls.
• Graceful Degradation and Uptime: In cases where the third-party API is experiencing downtime or delays, the API Data Proxy supports graceful degradation by serving previously stored data, even if slightly outdated. This allows core functionality to remain accessible, leaving it up to the client to determine whether to use this data or await a live update. In some scenarios, maintaining availability with slightly stale data enhances the user experience, preserving key functionality that might otherwise be disrupted. This approach ensures a resilient user experience, particularly for critical applications, and allows clients to optimize for continuity or precision based on their needs.

The API Data Proxy's goal is to manage API data rather than specify its internal use patterns within the company.

Consider the example of a request where real-time data may not be required:

GET /api/users/12345 HTTP/1.1
Host: internal-api.example.com
X-Allow-Stale: true
Authorization: Bearer <token>

HTTP/1.1 200 OK
Content-Type: application/json
X-Last-Fetched-SecondsAgo: 95

{
    "id": 12345,
    "reference": "bb5923c8-534b-468c-899f-a885357cddff",
    "firstName": "Chris",
    "lastName": "Haugner",
    "hairColor": "Brown",
    "hobbies": "Work"
}

Deep Dive: The Snapshot Approach in the API Data Proxy

The API Data Proxy uses a snapshot model to track changes in third-party API data over time. By storing versions of data in a "somewhat" structured format, it provides a comprehensive history of each data entity, enabling accurate audits and insights. Let's take a closer look at how this snapshotting works with an example.

JSON Example of API Data

Consider the following JSON response from an API representing user data:

{
    "id": 12345,
    "reference": "bb5923c8-534b-468c-899f-a885357cddff",
    "firstName": "Chris",
    "lastName": "Haugner",
    "hairColor": "Brown",
    "hobbies": "Work"
}

In this example, certain attributes (e.g., id and reference) are unique and immutable identifiers, while other fields (firstName, lastName, hairColor, hobbies) represent potentially mutable attributes that may change over time.

SQL Table Structure for Snapshot Storage

create table tbl_users (
    id_pk bigint serial primary key,
    reference varchar(256) not null,
    id varchar(256) not null,
    posted timestamp not null,
    deleted timestamp null,
    last_synced timestamp not null
);

create table tbl_user_snapshots (
    id_pk bigint serial primary key,
    user_fk bigint references tbl_users(id_pk),
    first_name varchar(max) not null,
    last_name varchar(max) not null,
    data text not null,
    posted timestamp not null
);

To capture and store this data, we use a database with two main tables: tbl_users and tbl_user_snapshots.

1. tbl_users: This table holds core metadata for each unique user instance, identifying immutable attributes extracted from the JSON, such as id and reference for identification in your API proxy DB. Additionally, it logs timestamps for tracking the creation (posted), deletion (i.e., when data is no longer available via the API, marked by deleted), and last sync time (last_synced) of each user entity.
2. tbl_user_snapshots: This table captures snapshots of the user's data over time, storing each version of the JSON data as a record in the (data) column. When a new JSON object is received, the proxy compares it to the latest snapshot in this table. If differences are found, a new record is added to maintain a historical record of changes.

This snapshot approach enables a complete and auditable history of each data entity, allowing precise tracking of when and how data attributes change over time. Snapshots are only created when data changes, ensuring efficient data usage patterns.

Yes, you guessed it, you will need to apply this DB table pairing concept for each entity that is managed on the API level. While this can lead to a significant number of tables, the process is straightforward and repetitive, minimizing cognitive complexity for each line of code.

During snapshotting you do not need to extract all attributes into SQL format; you can choose to extract only the attributes you care about or skip extraction altogether, acting on the raw JSON only. For example, in this case, attributes like hairColor and hobbies are ignored, as they are not relevant to our needs. However, uniqueness identification is required, so ensure you understand your patterns of key identification, whether it's single or composite, to ensure correct targeting of entities in your data model.

Beyond Data: Enhancing API Interaction

An API is more than just the data it provides; it also dictates how that data is requested and how clients interact with it. The API Data Proxy improves these interaction patterns by adapting to client needs while respecting the API's limitations. For example, if an API enforces strict throttling limits or restricts data to specific query parameters (e.g., single-day versus weekly date ranges), the proxy can transform client requests to better align with these limitations. By handling requests through batching or multi-threading, the proxy allows internal systems to access data more flexibly without being constrained by the API's restrictions, ultimately supporting a broader range of client interaction patterns.

Additional use cases include API Request Composition, where the proxy consolidates data typically split across multiple API requests and returns it as a single, unified response to your internal client.

Supporting Data Replication and Data Freshness Targets

With enhanced interaction control, the API Data Proxy moves beyond a simple reverse proxy, evolving into an autonomous data manager by incorporating what is often referred to as a Replication Operator. It proactively supports data synchronization by requesting updates from the API as needed, independent of any client request. This shift now transitions your data store into a true replica of a REST API and will allow the API Data Proxy to maintain data freshness targets and respond more effectively to changing data.

Additionally, the synchronization operator of the API Data Proxy can now establish and be governed by internal Service Level Indicators (SLIs) and Service Level Objectives (SLOs) for data age, enabling proactive control over data freshness.

This approach aligns closely with data replication and eventually consistent data models, where some delay is acceptable before source data fully synchronizes with its replica. By maintaining a defined lag time even when no internal requests are made (e.g., ensuring payment data for users is never more than 2 days old), the API Data Proxy makes data consistency goals explicit. This setup strikes a balance between timely updates and the operational load of real-time data synchronization, setting clear expectations for stakeholders who rely on accurate, up-to-date information. Remember, real-time requests will continue to proxy the third-party API and update data stores with snapshots as needed.

Use cases for optimizing these interaction patterns are extensive. For example, synchronization frequency can vary by need, such as triggering hourly updates for users with recent transaction requests, while users without balance changes in the last year may only require updates every two days.

API Data Proxy: A Step Towards Data Control

The API Data Proxy does more than replicate external data; it empowers you to transform and enhance this data, turning it into an asset that's fully aligned with your organization's needs. Over time, this proxy becomes a sophisticated data management tool, bridging external APIs with internal data flows and enabling real-time, customized data interaction patterns. For example:

• Operationalizing Data in an ODS with Change Data Capture (CDC): By applying CDC techniques, you can move from raw "API REST" data into fully operationalized entities within a new Operational Data Store (ODS). This structured "company-specific" data model, enriched with the relationships and attributes your business requires, powers downstream systems while adapting and optimizing the data for internal consumption.
• Stream Data Changes on the ODS level: Once the data is structured in the ODS, any changes triggered by the API Data Proxy can now publish complete business-entity-optimized changes from your ODS CDC to a Kafka topic (or similar event stream) for downstream consumers (not to be confused with the API Data Proxy CDC). This enables internal clients to react to optimized data changes in real time, eliminating the need to continuously poll or query the API (or ODS). Systems that rely on immediate updates can subscribe to these change events, creating a more efficient and responsive data ecosystem.

Through these enhancements, the API Data Proxy serves as a critical component within a larger data management suite, functioning as a consistent data source for advanced data handling and distribution. While the proxy itself remains true to its purpose of API data mirroring, it enables new capabilities within the broader ecosystem, such as feeding data into an ODS or supporting real-time updates. This setup allows your company to tailor data flows for optimal interaction, control, and responsiveness, while maintaining a single, reliable source for third-party data.

API Data Proxy vs. Egress Gateways: A Tale of Gateway Patterns

While the API Data Proxy provides robust data control, it's essential to distinguish it from an egress gateway, another tool commonly used for managing external data interactions. The difference between the two lies in the depth at which each handles data, similar to the distinction between Layer 3 and Layer 7 gateways in networking.

Egress Gateways: Protocol-Level Data Handling

In the data context, an egress gateway functions similarly to a Layer 3 gateway. It focuses on routing traffic based on network protocols rather than inspecting data content - operating as a content-unaware gateway. An egress gateway typically logs entire requests without interpreting the data entities within them. For instance, it might log a request to GET egress.company.cloud/thirdpartyapi/profile/123 but doesn't inspect specific fields like name or status. This approach makes it effective for routing and security at the protocol level but limits its ability to provide the data tracking needed for business decisions.

API Data Proxy: Entity-Level Data Awareness

The API Data Proxy, on the other hand, is similar to a Layer 7 gateway, designed for content-aware routing. It has a deep understanding of the data, storing and managing snapshots of each entity to monitor changes at the attribute level over time. Where an egress gateway provides routing and access control at the protocol level, the API Data Proxy delivers structured, actionable data insight at the entity level, supporting tracking, auditing, and change management in addition to routing.

Bonus points if you are still reading - while an egress gateway can be technically configured as a Layer 7 network load balancer with access to full request and response data, it treats this information as opaque - unaware of specific data entities. This comparison between egress gateways and API Data Proxies highlights the different purposes each serves. Both tools are valuable, but they operate at distinct levels within the data management stack.

Expanding the API Data Proxy: Client-Proxy Relationships and Broader Applications

Understanding the roles of third-party APIs, egress gateways, API Data Proxies, and ODSs reveals the interconnected relationships between clients and proxies as data moves through various systems. Each component acts as a link in the chain, ultimately connecting the end user to third-party data and operating as both client and proxy depending on its role within the data flow.

In this chain, the API Data Proxy acts as both a client to the egress gateway and an intermediary, controlling data flow between the external API and downstream services. The ODS, in turn, can act as a client or consumer to the API Data Proxy, which acts as the source for data derived from the third-party API. Each service within this tiered setup fulfills a specific, precise function, ensuring data consistency, accessibility, and reliability at each step. This layered relationship highlights how each component is integral to secure, consistent data access, linking clients to third-party data with surgical precision.

While often applied to third-party APIs, this model is equally valuable for internal/legacy APIs or even third-party apps you are managing yourself. Implementing the API Data Proxy as an intermediary for these internal systems can streamline access to outdated or complex APIs, which may otherwise be difficult to modify or consume directly. In addition, the API Data Proxy introduces audit tracking for internal systems, supporting entity-level data change monitoring, a function not typically available in legacy systems without this managed proxy layer.

Choosing the Right Tool for API Data Management

When integrating API data, third-party or internal, selecting the right tool depends on the importance of the data, the need for auditability, and the complexity of downstream API and/or data use. Here's a guide to help you determine when to use an egress gateway, an API Data Proxy, or an enhanced API Data Proxy with Change Data Capture (CDC) feeding an Operational Data Store (ODS).

• Egress for simple API integrations where data isn't critical to business operations.
• API Data Proxy for scenarios where API data is critical to decision-making or API access is difficult.
• API Data Proxy with CDC to an ODS for high-demand, complex data environments where multiple internal clients access data, originally sourced from a third party or internal API, or where data transformations are essential.

Given the below, we offer a variety of access patterns to our core License data, even enriched with identity reference data from our internal identity management platform.

Hmm, this does not quite seem right ... By the time you get to this point, you will have likely discovered that you need something else - more granular routing control for clients.

Future Directions: Orchestrating Data Access with Central and Domain-Specific Gateways

As businesses scale, managing diverse access patterns for shared data becomes increasingly complex. With numerous applications, often 20 or more, requiring various forms of access to the same core data, a structured approach to data orchestration is essential. The next evolution in API data management introduces a central API gateway that acts as a command center for determining what data each client needs, while directing those requests to specialized, domain-specific gateways that focus on how to best retrieve that data.

In this model, the central gateway functions as the primary entry point, handling client authentication and authorizing requests based on data needs. Once the "what" of the request is determined, it intelligently routes these requests to a business entity gateway. This domain-specific gateway, in turn, evaluates how best to fulfill the client's data requirements by directing the request to one of three backend layers - the ODS, API Data Proxy, or egress gateway - each serving the data in its own unique ways.

This layered gateway model ensures each client request is efficiently matched with the optimal data source. By distinguishing between the central gateway's role in identifying the "what" and domain-specific gateways' role in determining the "how", this architecture enhances control and data flexibility across applications.

Finally, let's not forget our streaming element.

Stay tuned for a future article exploring how a central API gateway with domain-focused routing can streamline data access across your organization, delivering a resilient, secure, and scalable data ecosystem - we may also talk about Kafka.

Conclusion: Combining Existing Patterns in a New Way

Although the specific term API Data Proxy is not widely used, the architectural pattern it embodies - capturing API data into internal data stores for resilience, consistency, and auditability - is based on well-established principles. Similar approaches can be seen in Operational Data Stores (ODS), Data Replicators, API caching, Egress Gateways, and Event Sourcing. However, what makes this approach novel and useful is how it combines these patterns into a cohesive solution for managing third-party API dependencies.

In many industries, similar problems are addressed using variations of this approach, but packaging it specifically as an API Data Proxy highlights its versatility. This solution is especially powerful when API data, third-party or not, drives critical business decisions that need to be auditable and available independently of the external API's reliability.

By formalizing this pattern into the concept of an API Data Proxy, we are bringing a fresh and practical approach to solving the challenges of third-party data management. This concept helps companies maintain control over the data that drives their operations, even when that data originates from external sources.

TLDR

The API Data Proxy is an architectural solution designed to capture and internalize critical third-party API data, ensuring businesses can manage and access this information on demand to drive key decisions. While challenges like API downtime, inconsistencies, and undocumented changes are concerns, the larger issue is maintaining data availability in a controlled, accessible way. The API Data Proxy addresses this by securely storing an internal mirror of external data, enabling continuous access, data consistency, and a fully auditable record. By preserving a structured history of data changes, this model supports reliable tracking and simplified auditing, capturing snapshots only when changes occur.

Moving beyond a standard reverse proxy, the API Data Proxy can proactively manage data synchronization, setting internal Service Level Objectives (SLOs) on data age to ensure reliable, up-to-date data for decision-making even in the absence of internal client API calls.

Though often associated with third-party APIs, this proxy model can also manage internal or legacy APIs, offering modernized access to complex or outdated systems without the need to modify the APIs themselves. When adding the proxy to internal or legacy systems, an additional benefit is the audit tracking now supported, which may not be natively available in the systems the proxy manages. This adaptability makes the API Data Proxy an essential layer within a resilient data ecosystem, linking tools and methods to ensure critical data remains accessible, auditable, and aligned with business needs.

By combining principles of data transformations, data replication, event sourcing, and caching, it provides controlled and reliable access to external data, fortifying organizations' ability to make data-driven decisions with confidence.