![Tamás Tárnok = [ C#, Sitecore, … ]](https://trnktms.com/wp-content/uploads/2021/11/pxart_white-small.png)
Since Sitecore Item As Resources (IAR) is available, it’s a common and best practice to pack all of your templates, renderings and all non-content items into a protobuf file, which can be read by Sitecore data providers. This post is meant to be a follow-up article of Generating your own custom IAR files of your Sitecore Items. Kamruz is mentioning the following in the section of generating IAR files from Unicorn items:
There seems to be a slight bug with the CLI that it cannot handle non-contiguous item paths. For example, if one config serialized a parent item path and only that parent item path, and another serialized a set of child folders to a different folder location then the CLI seems to think that there are items that are duplicated and serialized twice.
So this post is about to give a general solution for this problem and for a few others which I encountered while I was implementing IAR build in our build pipeline, which I’ll go through one-by-one and providing a fix one-by-one. In our case, multiple teams working on the same platform in the same repository and through the years caring about serialization lost and noone knows anymore what we need and what we don’t need. In an ideal world, serialization and configuration cleanup should be done first. In my case, we are talking more than ~20.000 items serialized. You can imagine how much time does it take to sync and publish all these items after each deployment (it took hours). The ultimate goal was to reduce this deployment time and risks, by using protobuf resource files. So if you are in a similar situation, using Sitecore IAR would be a quick win in your release pipeline. So, let’s dive into the issues with Unicorn yml files and folder structure.
Item duplicates
Sitecore CLI is not tolerant about having the same item ID in multiple yaml files. In our case this happens due to wrong configurations, as some root folders are added for multiple Unicorn configurations, therefore the same Sitecore item is serialized in multiple modules. If you think about it, it makes sense – how Sitecore CLI would know which one is the correct one if they are not in sync? The fix is to delete item duplicates and in my case, I keep the latest edited.
Non-contiguous item paths
This is the problem mentioned by Kamruz. The solution is to restructure files and folders to follow the item path structure in the file system as well. This also means to get rid of all “guid” folders, which is a workaround solution by Unicorn to able to handle file paths longer than 248 character. As part of this step, the empty “guid” folders are also deleted.
Invalid yml files
With Unicorn, we can also serialize roles, but those are not valid items for Sitecore IAR. So as part of the clean-up task, these are also identified and deleted.
+ Skipping content items
This is not bug, but an advice. I don’t recommend to put content items into protobuf, but just keep them in Unicorn if you really need it.
In Unicorn configs, we also could have modules where the Unicorn.Evaluators.NewItemOnlyEvaluator predicate is used, which means these items are only deployed once to the environments and then further changes are not applied on Unicorn sync. This is typically used for content and media items, but not only. So, I also extended the script to remove these items from the protobuf file, by reading all the configurations and remove the yml files from the Unicorn folder.
The script
Enough talk, here is the script. The script can be invoked like this.
.\build-iar.ps1
>> -UnicornItemsRootPath c:\Temp\Unicorn
>> -UnicornConfigsRootPath c:\Temp\App_Config
>> -OutputPath c:\Temp
Parameter explanation:
UnicornItemsRootPath: the root folder path of all the Unicorn yml filesUnicornConfigsRootPath: the explicit path of the\App_config\OutputPath: the path where the Sitecore IAR files has to be generated
| [CmdletBinding()] | |
| param ( | |
| [ValidateScript({ Test-Path $_ -PathType Container })] | |
| [string]$UnicornConfigsRootPath, | |
| [ValidateScript({ Test-Path $_ -PathType Container })] | |
| [string]$UnicornItemsRootPath, | |
| [string]$OutputPath | |
| ) | |
| Install-Module powershell-yaml -Scope CurrentUser -Force | |
| Import-Module powershell-yaml -Force | |
| #--- | |
| # COLLECT PATHS TO DELETE | |
| #--- | |
| Write-Host "[Cleanup 0/5] Collecting paths to delete..." -ForegroundColor Green | |
| $pathsToSkip = [System.Collections.ArrayList]@() | |
| Get-ChildItem -Path $UnicornConfigsRootPath -Recurse -Filter "*.config" | ForEach-Object { | |
| $fileContent = Get-Content -Path $_.FullName -Raw | |
| if (-not ($fileContent -like "*Unicorn.Evaluators.NewItemOnlyEvaluator*")) { | |
| return | |
| } | |
| [xml]$xmlContent = $fileContent | |
| foreach ($configuration in $xmlContent.configuration.sitecore.unicorn.configurations.configuration) { | |
| if (-not ($configuration.evaluator.type -like "*Unicorn.Evaluators.NewItemOnlyEvaluator*")) { | |
| continue | |
| } | |
| Write-Host "[Cleanup] Unicorn config name:"$configuration.name"-"$_.Name -ForegroundColor Yellow | |
| foreach ($include in $configuration.predicate.include) { | |
| Write-Host "[Cleanup]"$configuration.name $include.path -ForegroundColor Cyan | |
| $pathsToSkip.Add($include.path + "*") | Out-Null | |
| } | |
| } | |
| } | |
| #--- | |
| # CLEAN UP | |
| #--- | |
| Write-Host "[Cleanup 1/5] Collecting files to delete..." -ForegroundColor Green | |
| Get-ChildItem -Path $UnicornItemsRootPath -Recurse -Filter "*.yml" | ForEach-Object { | |
| $fileContent = Get-Content $_.FullName -Raw | |
| try { | |
| $yaml = ConvertFrom-Yaml $fileContent | |
| } | |
| catch { | |
| $yaml = $null | |
| } | |
| if ($yaml -eq $null -or (($pathsToSkip | Where-Object { $yaml.Path -like $_ }) -and $yaml.DB -eq "master")) { | |
| Write-Host "[Cleanup] Deleting" $_.FullName -ForegroundColor Cyan | |
| Remove-Item $_.FullName | |
| } | |
| } | |
| #--- | |
| # REMOVE DUPLICATES | |
| #--- | |
| $yamlFiles = Get-ChildItem -Path $UnicornItemsRootPath -Recurse -Filter '*.yml' | |
| $result = [System.Collections.ArrayList]@() | |
| $foundDuplicates = [System.Collections.ArrayList]@() | |
| $filesToDelete = [System.Collections.ArrayList]@() | |
| Write-Host "[Remove duplicates 2/5] Collecting duplicates..." -ForegroundColor Green | |
| foreach ($yamlFile in $yamlFiles) { | |
| $fileContent = Get-Content $yamlFile.FullName -Raw | |
| $yaml = ConvertFrom-Yaml $fileContent | |
| if ([string]::IsNullOrWhiteSpace($yaml.ID) -or -not [guid]::TryParse($yaml.ID, $([ref][guid]::Empty))) { | |
| $filesToDelete.Add([PSCustomObject]@{ Id = $null; File = $yamlFile }) | Out-Null | |
| Remove-Item -Path $yamlFile.FullName -Force | |
| } | |
| $duplicates = $result | Where-Object { $_.ID -eq $yaml.ID } | Sort-Object -Property LastWriteTime -Descending | Select-Object | |
| if ($duplicates.Count -gt 0) { | |
| $duplicatesWithId = $duplicates | Where-Object { $_.ID -eq $yaml.ID } | Select-Object | |
| if ($duplicatesWithId.Count -eq 1) { | |
| $foundDuplicates.Add([PSCustomObject]@{ Id = $duplicatesWithId[0].ID; File = $duplicatesWithId[0].File }) | Out-Null | |
| } | |
| $foundDuplicates.Add([PSCustomObject]@{ Id = $yaml.ID; File = $yamlFile }) | Out-Null | |
| } | |
| $result.Add([PSCustomObject]@{ Id = $yaml.ID; File = $yamlFile }) | Out-Null | |
| } | |
| Write-Host "[Remove duplicates 2/5] Found duplicates:" -ForegroundColor Green | |
| $foundDuplicates | Format-Table -Wrap -Property Id, @{ Label = "Path"; Expression = { $_.File.FullName } }, @{ Label = "LastWriteTimeUtc"; Expression = { $_.File.LastWriteTimeUtc } }, @{ Label = "CreationTimeUtc"; Expression = { $_.File.CreationTimeUtc } } | |
| $filesToKeep = $foundDuplicates | Group-Object -Property Id | Select-Object -Property @{ Name = "Id"; Expression = { $_.Name } }, @{ Name = "File"; Expression = { ($_.Group | Sort-Object -Descending -Property { if ($_.File.LastWriteTimeUtc -gt $_.File.CreationTimeUtc) { $_.File.LastWriteTimeUtc } else { $_.File.CreationTimeUtc } })[0].File } } | |
| Write-Host "[Remove duplicates 2/5] Files kept:" -ForegroundColor Green | |
| $filesToKeep | Format-Table -Wrap -Property Id, @{ Label = "Path"; Expression = { $_.File.FullName } }, @{ Label = "LastWriteTimeUtc"; Expression = { $_.File.LastWriteTimeUtc } }, @{ Label = "CreationTimeUtc"; Expression = { $_.File.CreationTimeUtc } } | |
| foreach ($fileToKeep in $filesToKeep) { | |
| $duplicates = $foundDuplicates | Where-Object { $_.Id -eq $fileToKeep.Id -and $_.File.FullName -ne $fileToKeep.File.FullName } | Select-Object | |
| foreach ($duplicate in $duplicates) { | |
| $filesToDelete.Add($duplicate) | Out-Null | |
| Remove-Item -Path $duplicate.File.FullName -Force | |
| } | |
| } | |
| Write-Host "[Remove duplicates 2/5] Files deleted:" -ForegroundColor Green | |
| $filesToDelete | Format-Table -Wrap -Property Id, @{ Label = "Path"; Expression = { $_.File.FullName } }, @{ Label = "LastWriteTimeUtc"; Expression = { $_.File.LastWriteTimeUtc } }, @{ Label = "CreationTimeUtc"; Expression = { $_.File.CreationTimeUtc } } | |
| #--- | |
| # RESTRUCTURE FILES | |
| #--- | |
| Write-Host "[Restructure files 3/5] Collecting folders to move..." -ForegroundColor Green | |
| $yamlFiles = Get-ChildItem -Path $UnicornItemsRootPath -Recurse -Filter "*.yml" | ForEach-Object { | |
| $fileContent = Get-Content $_.FullName -Raw | |
| $yaml = ConvertFrom-Yaml $fileContent | |
| $result = [PSCustomObject]@{ File = $_; Yaml = $yaml; Level = $yaml.Path.Split("/").Count } | |
| return $result | |
| } | Sort-Object -Property Level -Descending | |
| $allFolders = Get-ChildItem -Path $UnicornItemsRootPath -Recurse -Directory | |
| $foldersToMove = [System.Collections.ArrayList]@() | |
| $targets = [System.Collections.ArrayList]@() | |
| # Collect all folders which should move moved | |
| foreach ($folder in $allFolders) { | |
| $files = (Get-ChildItem -Filter "*.yml" -Path $folder.FullName) | |
| if ($files.Length -lt 1) { | |
| continue | |
| } | |
| $firstYamlFile = $files[0] | |
| $fileContent = Get-Content $firstYamlFile.FullName -Raw | |
| $yaml = ConvertFrom-Yaml $fileContent | |
| $foldersToMove.Add([PSCustomObject]@{ CurrentFolder = $folder; Parent = $yaml.Parent }) | Out-Null | |
| } | |
| # Collect where the folders has be moved | |
| foreach ($yamlFile in $yamlFiles) { | |
| if ([string]::IsNullOrWhiteSpace($yamlFile.Yaml.Path)) { | |
| continue | |
| } | |
| $foundChildren = $foldersToMove | Where-Object { $_.Parent -eq $yamlFile.Yaml.ID } | |
| if ($foundChildren.Length -lt 1) { | |
| continue | |
| } | |
| foreach ($foundChild in $foundChildren) { | |
| $targetFolderPath = $yamlFile.File.Directory.FullName + "\" + $yamlFile.File.Name.TrimEnd(".yml") | |
| if ($foundChild.CurrentFolder.FullName -like $targetFolderPath + "*") { | |
| continue | |
| } | |
| if (-not (Test-Path -Path $targetFolderPath)) { | |
| New-Item $targetFolderPath -ItemType Directory | Out-Null | |
| } | |
| $path = $foundChild.CurrentFolder.FullName + "\*" | |
| $filesToMove = Get-ChildItem -Path $path -Recurse | |
| $targetFolder = Get-Item $targetFolderPath | |
| $targets.Add([PSCustomObject]@{ Target = $targetFolder; FilesToMove = $filesToMove; SourcePath = $path }) | Out-Null | |
| } | |
| } | |
| Write-Host "[Restructure files 3/5] Move the collected folders..." -ForegroundColor Green | |
| # Move folders | |
| foreach ($target in $targets) { | |
| Write-Host $target.SourcePath -ForegroundColor Yellow | |
| Write-Host $target.Target.FullName -ForegroundColor Green | |
| Move-Item -Path $target.SourcePath -Destination $target.Target.FullName -Force | |
| Write-Host "---" | |
| } | |
| #--- | |
| # DELETE ID FOLDERS | |
| #--- | |
| Write-Host "[Delete GUID folders 4/5] Started..." -ForegroundColor Green | |
| $allFolders = Get-ChildItem -Path $UnicornItemsRootPath -Recurse -Directory | |
| foreach ($folder in $allFolders) { | |
| if ([string]::IsNullOrWhiteSpace($folder.Name) -or -not [guid]::TryParse($folder.Name, $([ref][guid]::Empty))) { | |
| continue | |
| } | |
| Remove-Item $folder.FullName -Recurse | |
| } | |
| #--- | |
| # BUILD IAR | |
| #--- | |
| Write-Host "[Generate Protobuf 5/5] Generate IAR file from Unicorn items..." -ForegroundColor Green | |
| dotnet nuget add source -n "Sitecore NuGet Feed" https://nuget.sitecore.com/resources/v3/index.json | |
| dotnet new tool-manifest --force | |
| dotnet tool install Sitecore.CLI --version 5.2.113 | |
| dotnet sitecore init | |
| if (-not [string]::IsNullOrEmpty($UnicornItemsRootPath)) { | |
| # Create the IAR files | |
| dotnet sitecore itemres unicorn -p $UnicornItemsRootPath -o $OutputPath"\unicorn" --overwrite --verbose | |
| } |
Tamás Tárnok = [ C#, Sitecore, ... ] 