Sitecore Headless Next.js – Generic component based external API response processing

Since Sitecore Headless and Composable DXP jumped in to our life, designing and architecting Sitecore solutions are changing. The integration layer is moving to the frontend layer and stays decoupled from Sitecore.

On the current project we are working on, almost all Sitecore components have to call different API routes to enrich the Sitecore component with data coming from these external sources. As it’s a requirement to almost all of our components, it makes sense to design a generic solution for serving the merged data to the frontend. I want to thank Richárd Palkó, we designed and implemented this generic solution together and thank you for guiding me on getting familiar with TypeScript, Next.js and React 🙏.

To lay down some rules, the responsibility between Sitecore and 3rd party APIs is the following:

Sitecore – responsible for layout definition and some marketing content (mostly text, images, links)
3rd party APIs – responsible for business content (price, restrictions, reservations, etc.)

I have to mention that on this project we are using Sitecore 10.2 with the Next.js SDK (SSR). In short, our idea is to extend the getServerSideProps response with the responses needed for our components.

Redis caching layer

Additionally we use Redis caching as well, because some of the 3rd party APIs are slow and we don’t have any influence on that. This layer is for optimization, if you have fast external APIs, you probably don’t need this. So we decided to use short period caching (5 minutes) and use Redis as a storage which is fairly easy to integrate with the node-redis package. This simple 2 diagram explains the caching mechanism.

Implementation

1. The `services.ts` – rule them all

The main implementation of calling the external APIs and Redis caching logic is in the services.ts class. Let me try to explain the different blocks of this file.

services
- This variable holds all the possible external service definitions in an array. As you can see it consist of a few optional properties, let me explain one-by-one
  - get – returns the actual endpoint which should consumed with all the dynamic parameters
  - getCallback – the callback function of the get function, just in case we need to merge API responses from 2 different sources, e.g. a Sitecore API response and 3rd party API response
  - getRedis – the function which gets called to try to get the given entry by the key from Redis
  - pushRedis – function which pushes the result of get to Redis
layoutServices
- This variable holds which Sitecore component need which external service. This is the base source for our implementation to find which component needs which external services to work properly. As a developer if I need another component which needs an external service, I need to extend this list and then I can use the external service response in the component.
getServicesByLayouts
- This is the method is responsible for collecting the external services for the components on the page. In our implementation it tries to find the actual components in the jss-content placeholder.
⚠ mapServicesToRequests
- This is the most important method as this holds the logic of calling the external APIs based on the flow diagram above.

	import { ComponentRendering, LayoutServicePageState } from '@sitecore-jss/sitecore-jss-nextjs';
	import axios, { AxiosError, AxiosRequestConfig, AxiosResponse } from 'axios';
	import { ParsedUrlQuery } from 'querystring';
	import { User } from '@/interfaces/user';
	import { mergeAxiosResponseArrays } from '@/lib/helpers';
	import RedisClient from '@/lib/redis-client';
	import { StyleguideSitecoreContextValue } from './component-props';

	interface Service {
	[key: string]: {
	get: (params: Record<string, unknown>, ...args: unknown[]) => string;
	getCallback?: (params: Record<string, unknown>, response: AxiosResponse, ...args: unknown[]) => string;
	getRedis?: (params: Record<string, unknown>, ...args: unknown[]) => Promise<AxiosResponse<unknown>>;
	pushRedis?: (params: Record<string, unknown>, ...args: unknown[]) => Promise<void>;
	};
	}

	interface LayoutWithService {
	[key: string]: { name: string; queryParams: string[]; pathMapping: string[]; contextName?: string; }[];
	}

	enum APISources { AIS_Products, AIS_Generic, Sitecore, Next }

	/**
	* Gets the required API host based on sitecore page state and API source
	*/
	const getAPIHost = (pageState: string, sourceName: APISources) => {
	const sources: { name: APISources; host?: string; mockHost?: string }[] = [
	{
	name: APISources.AIS_Products,
	host: process.env.AIS_HOST,
	mockHost: process.env.AIS_MOCK_HOST,
	},
	{
	name: APISources.AIS_Generic,
	host: process.env.AIS_BASE_URL,
	mockHost: process.env.AIS_MOCK_HOST,
	},
	{
	name: APISources.Sitecore,
	host: process.env.SITECORE_CUSTOMAPI_HOST,
	mockHost: process.env.SITECORE_USTOMAPI_MOCK_HOST,
	},
	{
	name: APISources.Next,
	host: `${process.env.PUBLIC_NEXT_API_URL}/api`,
	mockHost: process.env.AIS_MOCK_HOST,
	},
	];
	const selectedSource = sources.find((source) => source.name === sourceName);
	return pageState === LayoutServicePageState.Normal
	? selectedSource?.host
	: selectedSource?.mockHost;
	};

	/**
	* Gets response from Redis
	*/
	const getRedis = async (key: string, pageState: string, contextName?: string) => {
	if (pageState !== LayoutServicePageState.Normal) {
	return null;
	}

	const client = RedisClient.getRedisClient();
	const value = await client.get(key);
	const parsedValue = JSON.parse(value ?? 'null');
	return contextName ? { [contextName]: parsedValue } : parsedValue;
	};

	/**
	* Pushes value to Redis
	*/
	export const pushRedis = async (
	key: string,
	value: string,
	pageState: string,
	useLongExpiration = false,
	) => {
	const expirationTime = process.env[useLongExpiration ? 'REDIS_EXPIRE_LONG' : 'REDIS_EXPIRE'] ?? '60';
	if (pageState !== LayoutServicePageState.Normal) {
	return;
	}

	const client = RedisClient.getRedisClient();
	await client.set(key, value, { EX: Number.parseInt(expirationTime), });
	};

	// List of each service and its API endpoint
	export const services: Service = {
	products: {
	get: (params: { userId: string; pageState: string }) =>
	`${getAPIHost(params.pageState, APISources.AIS_Products)}/user/${
	params.userId
	}/products`,
	getRedis: ({ userId, pageState }: { userId: string; pageState: string }) =>
	getRedis(`${userId}.products`, pageState),
	pushRedis: ({ userId, value, pageState,
	}: {
	userId: string;
	value: string;
	pageState: string;
	}) => pushRedis(`${userId}.products`, value, pageState),
	},
	reservations: {
	get: (params: { userId: string; pageState: string }) =>
	`${getAPIHost(params.pageState, APISources.Next)}/user/${params.userId}/reservations`,
	getRedis: ({ userId, pageState }: { userId: string; pageState: string }) =>
	getRedis(`${userId}.reservations`, pageState),
	pushRedis: ({ userId, value, pageState }: {
	userId: string;
	value: string;
	pageState: string;
	}) => pushRedis(`${userId}.reservations`, value, pageState),
	},
	countries: {
	get: (params: { pageState: string }) =>
	`${getAPIHost(params.pageState, APISources.AIS_Generic)}/location/v1/countries`,
	getRedis: ({ pageState, contextName }: { pageState: string; contextName: string }) =>
	getRedis('countries', pageState, contextName),
	pushRedis: ({ value, pageState }: { value: string; pageState: string }) =>
	pushRedis('countries', value, pageState, true),
	},
	};

	// List of possible layouts with their associated services
	export const layoutServices: LayoutWithService = {
	Products: [{ name: 'products', queryParams: [], pathMapping: [] }],
	ProductDetails: [{ name: 'products', queryParams: [], pathMapping: [] }],
	ProductFinder: [
	{ name: 'reservations', queryParams: [], pathMapping: [] },
	{ name: 'products', queryParams: [], pathMapping: [] },
	],
	};

	/**
	* Maps the Sitecore components (or layouts) to a list
	*/
	export const getLayoutsFromSitecore = (
	sitecoreContext: StyleguideSitecoreContextValue \| null,
	): string[] => {
	if (!sitecoreContext) {
	return [];
	}
	const jssMain = sitecoreContext.route.placeholders['jss-main'].filter((rendering) => (rendering as ComponentRendering)?.componentName)[0] as ComponentRendering;
	const jssContent = jssMain?.placeholders?.['jss-content'] as ComponentRendering[];
	return jssContent?.map((content) => content.componentName);
	};

	/**
	* Maps the services needs to be called to a list
	*/
	export const getServicesByLayouts = (sitecoreContext: StyleguideSitecoreContextValue \| null) => {
	const layouts = getLayoutsFromSitecore(sitecoreContext);
	return layouts?.map((layout) => layoutServices[layout]).flat().filter(Boolean);
	};

	/**
	* Maps browser path by given list of path mapping to object
	*/
	export const getPathMappingFromService = (path: string[], pathMapping: string[]) => {
	return (
	pathMapping?.reduce((prev, currentPath, i) => {
	prev[currentPath] = path[i];
	return prev;
	}, {} as Record<string, string>) \|\| {}
	);
	};

	/**
	*
	* @param sitecoreContext The actual rendering context of Sitecore
	*/
	export const mapServicesToRequests = (
	sitecoreContext: StyleguideSitecoreContextValue \| null,
	queryParams: ParsedUrlQuery,
	user?: User,
	): Promise<AxiosResponse<unknown>>[] => {
	const servicesToCall = getServicesByLayouts(sitecoreContext) \|\| [];
	const { path, ...getParams } = queryParams;
	const params = getParams as Record<string, string>;
	params.pageState = sitecoreContext?.pageState?.toString() ?? '';

	return servicesToCall.map(async (service) => {
	const mappedPath = getPathMappingFromService(path as string[], service.pathMapping);

	const foundService = services[service.name];
	if (foundService?.getRedis) {
	// 1. try to load the response from Redis
	let response: AxiosResponse<unknown> \| null = null;
	const redisCall = foundService
	?.getRedis({ ...params, ...user, ...mappedPath, contextName: service?.contextName })
	.then((redisResponse) => {
	response = redisResponse;
	return new Promise<AxiosResponse<unknown>>((resolve) =>
	resolve({ data: redisResponse } as unknown as AxiosResponse<unknown>),
	);
	});

	await redisCall;

	if (response) {
	// return the response from Redis if it has a response
	return new Promise<AxiosResponse<unknown>>((resolve) =>
	resolve({ data: response } as unknown as AxiosResponse<unknown>),
	);
	}
	}

	return axios
	.get(
	// 2. load the response from AIS
	foundService.get({ ...params, ...user, ...mappedPath }),
	{ headers: { contextName: service?.contextName \|\| null } },
	)
	.then((axiosResponse) => {
	// 3. load and merge response from callback, if any
	if (foundService.getCallback && axiosResponse.status === 200 && axiosResponse.data) {
	return axios
	.get(
	foundService.getCallback(
	{
	...params,
	...user,
	...mappedPath,
	},
	axiosResponse,
	),
	{ headers: { contextName: service?.contextName \|\| null } },
	)
	.then((callbackResponse) => {
	if (callbackResponse.status === 200 && callbackResponse.data) {
	axiosResponse.data = {
	[service.name]: mergeAxiosResponseArrays(axiosResponse, callbackResponse),
	};
	}
	return axiosResponse;
	});
	}
	return axiosResponse;
	})
	.then((axiosResponse) => {
	// 4. push the response from AIS to Redis
	if (axiosResponse.status === 200 && axiosResponse.data) {
	params.value = JSON.stringify(axiosResponse.data);
	foundService?.pushRedis?.({ ...params, ...user, ...mappedPath });
	}
	return axiosResponse;
	});
	});
	};

	/**
	* Handles errors came from API calls
	*/
	export const handleGetServerSidePropsDataFetchError = (error: Error \| AxiosError) => {
	if (axios.isAxiosError(error)) {
	console.error(error);
	return {
	redirect: {
	permanent: false,
	destination: `/${error?.response?.status \|\| 500}`,
	},
	props: {},
	};
	} else {
	return {
	redirect: {
	permanent: false,
	destination: '/500',
	},
	};
	}
	};

	/**
	* Maps a list of API response data to an object
	*/
	export const reduceAxiosResponses = <T>(responses: AxiosResponse<T>[]) =>
	responses
	.map(({ data, config }) => [data, config])
	.reduce((allData, [data, config]) => {
	const contextName = (config as AxiosRequestConfig)?.headers?.contextName;
	// Wrap data in a field for easier access in state manager if contextName is provided
	const transformedData = contextName ? { [contextName]: data } : data;
	return { ...allData, ...transformedData };
	}, {});

view raw services.ts hosted with ❤ by GitHub

2. The `redis-client.ts` – make it fast

The redis-client is just a wrapper around node-redis. This class is just to initiate the Redis client with some basic settings.

	import { createClient } from 'redis';

	class RedisClient {
	private static redisClient: ReturnType<typeof createClient>;

	private static initRedisClient() {
	const redisClient = createClient({
	url: process.env.REDIS_URL,
	password: process.env.REDIS_PASSWORD,
	socket: {
	tls: !process.env.REDIS_URL?.includes?.('localhost'),
	connectTimeout: 3000,
	reconnectStrategy: () => 360000,
	},
	});

	redisClient.on('connect', () => console.log('Redis client is connected...'));
	redisClient.on('ready', () => console.log('Redis client is ready...'));
	redisClient.on('error', (err) => console.log('Redis client error occured', err));
	redisClient.on('end', () => console.log('Redis client is disconnected'));
	process.on('SIGINT', () => redisClient.quit());
	redisClient.connect();
	return redisClient;
	}

	public static getRedisClient(): ReturnType<typeof createClient> {
	if (!RedisClient.redisClient) {
	RedisClient.redisClient = RedisClient.initRedisClient();
	}
	return RedisClient.redisClient;
	}
	}

	export default RedisClient;

view raw redis-client.ts hosted with ❤ by GitHub

3. The `[[…path]].tsx` – pass to components

The next bigger change is in the [[...path]].tsx which is a file you get generated when you create the Next.js project with the Sitecore JSS CLI. As you would guess we had to extend it with the responses from the external APIs, to able to get access to the data in the component level. As mentioned above, for this we had to extend getServerSideProps to extend the returned object. Here is the call to the already mentioned mapServicesToRequests method which returns which external APIs needs to be called for the components on the requested page. Then with the Promise.all it gets actually called and based on the logic the data returned from the external APIs or from Redis in case of a cache hit.

	import { ComponentPropsContext, handleEditorFastRefresh, SitecoreContext } from '@sitecore-jss/sitecore-jss-nextjs';
	import { GetServerSideProps } from 'next';
	import { getSession } from 'next-auth/react';
	import { createContext, useEffect } from 'react';
	import Layout from 'src/Layout';
	import NotFound from 'src/NotFound';
	import { componentFactory } from 'temp/componentFactory';
	import { Product } from '@/interfaces/product';
	import { Country } from '@/interfaces/address';
	import { GetReservationsResponse } from '@/interfaces/booking';
	import { StyleguideSitecoreContextValue } from '@/lib/component-props';
	import { SitecorePageProps } from '@/lib/page-props';
	import { sitecorePagePropsFactory } from '@/lib/page-props-factory';
	import { handleGetServerSidePropsDataFetchError, mapServicesToRequests, reduceAxiosResponses } from '@/lib/services';

	interface SitecorePagePropsWithServiceData extends SitecorePageProps {
	data: Partial<ExtendedContext>;
	}

	interface ExtendedContext {
	products?: Product[];
	reservations?: GetReservationsResponse[];
	countries?: Country[];
	}

	export interface User {
	userId: string;
	}

	export const AppContext = createContext<ExtendedContext>({});

	const SitecorePage = ({ notFound, componentProps, sitecoreContext, data }: SitecorePagePropsWithServiceData): JSX.Element => {
	useEffect(() => {
	// Since Sitecore editors do not support Fast Refresh, need to refresh EE chromes after Fast Refresh finished
	handleEditorFastRefresh();
	}, []);

	if (notFound \|\| !sitecoreContext?.route) {
	// Shouldn't hit this (as long as 'notFound' is being returned below), but just to be safe
	return <NotFound />;
	}

	return (
	<ComponentPropsContext value={componentProps}>
	<SitecoreContext<StyleguideSitecoreContextValue> componentFactory={componentFactory} context={sitecoreContext}>
	<AppContext.Provider value={data as ExtendedContext}>
	<Layout />
	</AppContext.Provider>
	</SitecoreContext>
	</ComponentPropsContext>
	);
	};

	// This function gets called at request time on server-side.
	export const getServerSideProps: GetServerSideProps = async (context) => {
	const session = (await getSession(context));
	const props = (await sitecorePagePropsFactory.create(context)) as SitecorePagePropsWithServiceData;

	if (props?.sitecoreContext?.route?.redirect) {
	return {
	redirect: props.sitecoreContext.route.redirect \|\| '/404',
	props: {},
	};
	}

	// Fetch and map AIS service data to props for each layout
	try {
	const user : User = { userId: session?.client.userId ?? '' };
	const serviceRequests = mapServicesToRequests(props.sitecoreContext, context.query, user);
	const resolvedRequests = await Promise.all(serviceRequests);
	props.data = reduceAxiosResponses(resolvedRequests);
	} catch (e) {
	return handleGetServerSidePropsDataFetchError(e as Error);
	}

	return {
	props: { ...props, session },
	notFound: props.notFound, // Returns custom 404 page with a status code of 404 when true
	};
	};

	export default SitecorePage;

view raw [[...path]].tsx hosted with ❤ by GitHub

4. A sample component

As last bit here is a simplified example, how the external API data can be used on a single Sitecore component, using the AppContext.

	import { AppContext } from '@/pages/[[...path]]';
	import { Box } from '@mui/material';
	import { useContext } from 'react';

	export const Products = (): JSX.Element => {
	const { products: contextProducts } = useContext(AppContext);
	return (<Box>{contextProducts?.map(p => p.Name).concat()}</Box>);
	};

	export default Products;

view raw Products.tsx hosted with ❤ by GitHub

One thought on “Sitecore Headless Next.js – Generic component based external API response processing”

Azam Shaikh says:

January 26, 2023 at 16:37

Hi

Excellent article with real case scenario of external data. By any chance do you have this sample on git? Would love to see it in action. If you can add a new page too with the sitecore and external data components that will be wonderful.

Regards

LikeLike

Redis caching layer

Implementation

1. The services.ts – rule them all

2. The redis-client.ts – make it fast

3. The [[…path]].tsx – pass to components

4. A sample component

Share this:

One thought on “Sitecore Headless Next.js – Generic component based external API response processing”

Leave a Reply Cancel reply

1. The `services.ts` – rule them all

2. The `redis-client.ts` – make it fast

3. The `[[…path]].tsx` – pass to components