Sitecore Headless Next.js – Generic component based external API response processing

Since Sitecore Headless and Composable DXP jumped in to our life, designing and architecting Sitecore solutions are changing. The integration layer is moving to the frontend layer and stays decoupled from Sitecore.

On the current project we are working on, almost all Sitecore components have to call different API routes to enrich the Sitecore component with data coming from these external sources. As it’s a requirement to almost all of our components, it makes sense to design a generic solution for serving the merged data to the frontend. I want to thank Richárd Palkó, we designed and implemented this generic solution together and thank you for guiding me on getting familiar with TypeScript, Next.js and React 🙏.

To lay down some rules, the responsibility between Sitecore and 3rd party APIs is the following:

  • Sitecore – responsible for layout definition and some marketing content (mostly text, images, links)
  • 3rd party APIs – responsible for business content (price, restrictions, reservations, etc.)

I have to mention that on this project we are using Sitecore 10.2 with the Next.js SDK (SSR). In short, our idea is to extend the getServerSideProps response with the responses needed for our components.

Redis caching layer

Additionally we use Redis caching as well, because some of the 3rd party APIs are slow and we don’t have any influence on that. This layer is for optimization, if you have fast external APIs, you probably don’t need this. So we decided to use short period caching (5 minutes) and use Redis as a storage which is fairly easy to integrate with the node-redis package. This simple 2 diagram explains the caching mechanism.

1. Redis Cache miss
2. Redis Cache hit

Implementation

1. The services.ts – rule them all

The main implementation of calling the external APIs and Redis caching logic is in the services.ts class. Let me try to explain the different blocks of this file.

  • services
    • This variable holds all the possible external service definitions in an array. As you can see it consist of a few optional properties, let me explain one-by-one
      • get – returns the actual endpoint which should consumed with all the dynamic parameters
      • getCallback – the callback function of the get function, just in case we need to merge API responses from 2 different sources, e.g. a Sitecore API response and 3rd party API response
      • getRedis – the function which gets called to try to get the given entry by the key from Redis
      • pushRedis – function which pushes the result of get to Redis
  • layoutServices
    • This variable holds which Sitecore component need which external service. This is the base source for our implementation to find which component needs which external services to work properly. As a developer if I need another component which needs an external service, I need to extend this list and then I can use the external service response in the component.
  • getServicesByLayouts
    • This is the method is responsible for collecting the external services for the components on the page. In our implementation it tries to find the actual components in the jss-content placeholder.
  • mapServicesToRequests
    • This is the most important method as this holds the logic of calling the external APIs based on the flow diagram above.
import { ComponentRendering, LayoutServicePageState } from '@sitecore-jss/sitecore-jss-nextjs';
import axios, { AxiosError, AxiosRequestConfig, AxiosResponse } from 'axios';
import { ParsedUrlQuery } from 'querystring';
import { User } from '@/interfaces/user';
import { mergeAxiosResponseArrays } from '@/lib/helpers';
import RedisClient from '@/lib/redis-client';
import { StyleguideSitecoreContextValue } from './component-props';
interface Service {
[key: string]: {
get: (params: Record<string, unknown>, ...args: unknown[]) => string;
getCallback?: (params: Record<string, unknown>, response: AxiosResponse, ...args: unknown[]) => string;
getRedis?: (params: Record<string, unknown>, ...args: unknown[]) => Promise<AxiosResponse<unknown>>;
pushRedis?: (params: Record<string, unknown>, ...args: unknown[]) => Promise<void>;
};
}
interface LayoutWithService {
[key: string]: { name: string; queryParams: string[]; pathMapping: string[]; contextName?: string; }[];
}
enum APISources { AIS_Products, AIS_Generic, Sitecore, Next }
/**
* Gets the required API host based on sitecore page state and API source
*/
const getAPIHost = (pageState: string, sourceName: APISources) => {
const sources: { name: APISources; host?: string; mockHost?: string }[] = [
{
name: APISources.AIS_Products,
host: process.env.AIS_HOST,
mockHost: process.env.AIS_MOCK_HOST,
},
{
name: APISources.AIS_Generic,
host: process.env.AIS_BASE_URL,
mockHost: process.env.AIS_MOCK_HOST,
},
{
name: APISources.Sitecore,
host: process.env.SITECORE_CUSTOMAPI_HOST,
mockHost: process.env.SITECORE_USTOMAPI_MOCK_HOST,
},
{
name: APISources.Next,
host: `${process.env.PUBLIC_NEXT_API_URL}/api`,
mockHost: process.env.AIS_MOCK_HOST,
},
];
const selectedSource = sources.find((source) => source.name === sourceName);
return pageState === LayoutServicePageState.Normal
? selectedSource?.host
: selectedSource?.mockHost;
};
/**
* Gets response from Redis
*/
const getRedis = async (key: string, pageState: string, contextName?: string) => {
if (pageState !== LayoutServicePageState.Normal) {
return null;
}
const client = RedisClient.getRedisClient();
const value = await client.get(key);
const parsedValue = JSON.parse(value ?? 'null');
return contextName ? { [contextName]: parsedValue } : parsedValue;
};
/**
* Pushes value to Redis
*/
export const pushRedis = async (
key: string,
value: string,
pageState: string,
useLongExpiration = false,
) => {
const expirationTime = process.env[useLongExpiration ? 'REDIS_EXPIRE_LONG' : 'REDIS_EXPIRE'] ?? '60';
if (pageState !== LayoutServicePageState.Normal) {
return;
}
const client = RedisClient.getRedisClient();
await client.set(key, value, { EX: Number.parseInt(expirationTime), });
};
// List of each service and its API endpoint
export const services: Service = {
products: {
get: (params: { userId: string; pageState: string }) =>
`${getAPIHost(params.pageState, APISources.AIS_Products)}/user/${
params.userId
}/products`,
getRedis: ({ userId, pageState }: { userId: string; pageState: string }) =>
getRedis(`${userId}.products`, pageState),
pushRedis: ({ userId, value, pageState,
}: {
userId: string;
value: string;
pageState: string;
}) => pushRedis(`${userId}.products`, value, pageState),
},
reservations: {
get: (params: { userId: string; pageState: string }) =>
`${getAPIHost(params.pageState, APISources.Next)}/user/${params.userId}/reservations`,
getRedis: ({ userId, pageState }: { userId: string; pageState: string }) =>
getRedis(`${userId}.reservations`, pageState),
pushRedis: ({ userId, value, pageState }: {
userId: string;
value: string;
pageState: string;
}) => pushRedis(`${userId}.reservations`, value, pageState),
},
countries: {
get: (params: { pageState: string }) =>
`${getAPIHost(params.pageState, APISources.AIS_Generic)}/location/v1/countries`,
getRedis: ({ pageState, contextName }: { pageState: string; contextName: string }) =>
getRedis('countries', pageState, contextName),
pushRedis: ({ value, pageState }: { value: string; pageState: string }) =>
pushRedis('countries', value, pageState, true),
},
};
// List of possible layouts with their associated services
export const layoutServices: LayoutWithService = {
Products: [{ name: 'products', queryParams: [], pathMapping: [] }],
ProductDetails: [{ name: 'products', queryParams: [], pathMapping: [] }],
ProductFinder: [
{ name: 'reservations', queryParams: [], pathMapping: [] },
{ name: 'products', queryParams: [], pathMapping: [] },
],
};
/**
* Maps the Sitecore components (or layouts) to a list
*/
export const getLayoutsFromSitecore = (
sitecoreContext: StyleguideSitecoreContextValue | null,
): string[] => {
if (!sitecoreContext) {
return [];
}
const jssMain = sitecoreContext.route.placeholders['jss-main'].filter((rendering) => (rendering as ComponentRendering)?.componentName)[0] as ComponentRendering;
const jssContent = jssMain?.placeholders?.['jss-content'] as ComponentRendering[];
return jssContent?.map((content) => content.componentName);
};
/**
* Maps the services needs to be called to a list
*/
export const getServicesByLayouts = (sitecoreContext: StyleguideSitecoreContextValue | null) => {
const layouts = getLayoutsFromSitecore(sitecoreContext);
return layouts?.map((layout) => layoutServices[layout]).flat().filter(Boolean);
};
/**
* Maps browser path by given list of path mapping to object
*/
export const getPathMappingFromService = (path: string[], pathMapping: string[]) => {
return (
pathMapping?.reduce((prev, currentPath, i) => {
prev[currentPath] = path[i];
return prev;
}, {} as Record<string, string>) || {}
);
};
/**
*
* @param sitecoreContext The actual rendering context of Sitecore
*/
export const mapServicesToRequests = (
sitecoreContext: StyleguideSitecoreContextValue | null,
queryParams: ParsedUrlQuery,
user?: User,
): Promise<AxiosResponse<unknown>>[] => {
const servicesToCall = getServicesByLayouts(sitecoreContext) || [];
const { path, ...getParams } = queryParams;
const params = getParams as Record<string, string>;
params.pageState = sitecoreContext?.pageState?.toString() ?? '';
return servicesToCall.map(async (service) => {
const mappedPath = getPathMappingFromService(path as string[], service.pathMapping);
const foundService = services[service.name];
if (foundService?.getRedis) {
// 1. try to load the response from Redis
let response: AxiosResponse<unknown> | null = null;
const redisCall = foundService
?.getRedis({ ...params, ...user, ...mappedPath, contextName: service?.contextName })
.then((redisResponse) => {
response = redisResponse;
return new Promise<AxiosResponse<unknown>>((resolve) =>
resolve({ data: redisResponse } as unknown as AxiosResponse<unknown>),
);
});
await redisCall;
if (response) {
// return the response from Redis if it has a response
return new Promise<AxiosResponse<unknown>>((resolve) =>
resolve({ data: response } as unknown as AxiosResponse<unknown>),
);
}
}
return axios
.get(
// 2. load the response from AIS
foundService.get({ ...params, ...user, ...mappedPath }),
{ headers: { contextName: service?.contextName || null } },
)
.then((axiosResponse) => {
// 3. load and merge response from callback, if any
if (foundService.getCallback && axiosResponse.status === 200 && axiosResponse.data) {
return axios
.get(
foundService.getCallback(
{
...params,
...user,
...mappedPath,
},
axiosResponse,
),
{ headers: { contextName: service?.contextName || null } },
)
.then((callbackResponse) => {
if (callbackResponse.status === 200 && callbackResponse.data) {
axiosResponse.data = {
[service.name]: mergeAxiosResponseArrays(axiosResponse, callbackResponse),
};
}
return axiosResponse;
});
}
return axiosResponse;
})
.then((axiosResponse) => {
// 4. push the response from AIS to Redis
if (axiosResponse.status === 200 && axiosResponse.data) {
params.value = JSON.stringify(axiosResponse.data);
foundService?.pushRedis?.({ ...params, ...user, ...mappedPath });
}
return axiosResponse;
});
});
};
/**
* Handles errors came from API calls
*/
export const handleGetServerSidePropsDataFetchError = (error: Error | AxiosError) => {
if (axios.isAxiosError(error)) {
console.error(error);
return {
redirect: {
permanent: false,
destination: `/${error?.response?.status || 500}`,
},
props: {},
};
} else {
return {
redirect: {
permanent: false,
destination: '/500',
},
};
}
};
/**
* Maps a list of API response data to an object
*/
export const reduceAxiosResponses = <T>(responses: AxiosResponse<T>[]) =>
responses
.map(({ data, config }) => [data, config])
.reduce((allData, [data, config]) => {
const contextName = (config as AxiosRequestConfig)?.headers?.contextName;
// Wrap data in a field for easier access in state manager if contextName is provided
const transformedData = contextName ? { [contextName]: data } : data;
return { ...allData, ...transformedData };
}, {});
view raw services.ts hosted with ❤ by GitHub

2. The redis-client.ts – make it fast

The redis-client is just a wrapper around node-redis. This class is just to initiate the Redis client with some basic settings.

import { createClient } from 'redis';
class RedisClient {
private static redisClient: ReturnType<typeof createClient>;
private static initRedisClient() {
const redisClient = createClient({
url: process.env.REDIS_URL,
password: process.env.REDIS_PASSWORD,
socket: {
tls: !process.env.REDIS_URL?.includes?.('localhost'),
connectTimeout: 3000,
reconnectStrategy: () => 360000,
},
});
redisClient.on('connect', () => console.log('Redis client is connected...'));
redisClient.on('ready', () => console.log('Redis client is ready...'));
redisClient.on('error', (err) => console.log('Redis client error occured', err));
redisClient.on('end', () => console.log('Redis client is disconnected'));
process.on('SIGINT', () => redisClient.quit());
redisClient.connect();
return redisClient;
}
public static getRedisClient(): ReturnType<typeof createClient> {
if (!RedisClient.redisClient) {
RedisClient.redisClient = RedisClient.initRedisClient();
}
return RedisClient.redisClient;
}
}
export default RedisClient;
view raw redis-client.ts hosted with ❤ by GitHub

3. The [[…path]].tsx – pass to components

The next bigger change is in the [[...path]].tsx which is a file you get generated when you create the Next.js project with the Sitecore JSS CLI. As you would guess we had to extend it with the responses from the external APIs, to able to get access to the data in the component level. As mentioned above, for this we had to extend getServerSideProps to extend the returned object. Here is the call to the already mentioned mapServicesToRequests method which returns which external APIs needs to be called for the components on the requested page. Then with the Promise.all it gets actually called and based on the logic the data returned from the external APIs or from Redis in case of a cache hit.

import { ComponentPropsContext, handleEditorFastRefresh, SitecoreContext } from '@sitecore-jss/sitecore-jss-nextjs';
import { GetServerSideProps } from 'next';
import { getSession } from 'next-auth/react';
import { createContext, useEffect } from 'react';
import Layout from 'src/Layout';
import NotFound from 'src/NotFound';
import { componentFactory } from 'temp/componentFactory';
import { Product } from '@/interfaces/product';
import { Country } from '@/interfaces/address';
import { GetReservationsResponse } from '@/interfaces/booking';
import { StyleguideSitecoreContextValue } from '@/lib/component-props';
import { SitecorePageProps } from '@/lib/page-props';
import { sitecorePagePropsFactory } from '@/lib/page-props-factory';
import { handleGetServerSidePropsDataFetchError, mapServicesToRequests, reduceAxiosResponses } from '@/lib/services';
interface SitecorePagePropsWithServiceData extends SitecorePageProps {
data: Partial<ExtendedContext>;
}
interface ExtendedContext {
products?: Product[];
reservations?: GetReservationsResponse[];
countries?: Country[];
}
export interface User {
userId: string;
}
export const AppContext = createContext<ExtendedContext>({});
const SitecorePage = ({ notFound, componentProps, sitecoreContext, data }: SitecorePagePropsWithServiceData): JSX.Element => {
useEffect(() => {
// Since Sitecore editors do not support Fast Refresh, need to refresh EE chromes after Fast Refresh finished
handleEditorFastRefresh();
}, []);
if (notFound || !sitecoreContext?.route) {
// Shouldn't hit this (as long as 'notFound' is being returned below), but just to be safe
return <NotFound />;
}
return (
<ComponentPropsContext value={componentProps}>
<SitecoreContext<StyleguideSitecoreContextValue> componentFactory={componentFactory} context={sitecoreContext}>
<AppContext.Provider value={data as ExtendedContext}>
<Layout />
</AppContext.Provider>
</SitecoreContext>
</ComponentPropsContext>
);
};
// This function gets called at request time on server-side.
export const getServerSideProps: GetServerSideProps = async (context) => {
const session = (await getSession(context));
const props = (await sitecorePagePropsFactory.create(context)) as SitecorePagePropsWithServiceData;
if (props?.sitecoreContext?.route?.redirect) {
return {
redirect: props.sitecoreContext.route.redirect || '/404',
props: {},
};
}
// Fetch and map AIS service data to props for each layout
try {
const user : User = { userId: session?.client.userId ?? '' };
const serviceRequests = mapServicesToRequests(props.sitecoreContext, context.query, user);
const resolvedRequests = await Promise.all(serviceRequests);
props.data = reduceAxiosResponses(resolvedRequests);
} catch (e) {
return handleGetServerSidePropsDataFetchError(e as Error);
}
return {
props: { ...props, session },
notFound: props.notFound, // Returns custom 404 page with a status code of 404 when true
};
};
export default SitecorePage;
view raw [[...path]].tsx hosted with ❤ by GitHub

4. A sample component

As last bit here is a simplified example, how the external API data can be used on a single Sitecore component, using the AppContext.

import { AppContext } from '@/pages/[[...path]]';
import { Box } from '@mui/material';
import { useContext } from 'react';
export const Products = (): JSX.Element => {
const { products: contextProducts } = useContext(AppContext);
return (<Box>{contextProducts?.map(p => p.Name).concat()}</Box>);
};
export default Products;
view raw Products.tsx hosted with ❤ by GitHub

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s