Introduction
It is important for many development jobs to be able to look at the differences between PowerShell objects.
It might be that you are checking how Windows processes change over time: Perhaps you are monitoring various signs of stress on a server but are at the exploratory stage, where you need to see the metrics that seem to be correlating: You might be wanting to make a high-level check to changes to a database by comparing the metadata to see what has changed. I use it to get a ‘narrative of changes’ in databases under development, and roughly when they happened.
One of the most common things you need to be able to do when developing stuff is to be able to do automated unit tests. This means detecting automatically the actual output with the correct output.
Whatever your development methodology, you need to make changes lightning fast, and the easiest way of doing that is to test frequently. If you are driving this work with PowerShell, which works well, you’ll want to compare the actual results of a process with the expected results. You’re keen to see what’s changed but will often have no idea what to look for beforehand. You need the broad view.
Fine. To do this, you need something that can tell you the differences between two objects. Yes, there is already a cmdlet to do that called Compare-Object. It is useful and ingenious, and works well for what is does, but it doesn’t do enough for our purposes. Let’s run through a few examples, just to explain why I need more for many of the things I do.
Using PowerShell’s Compare-Objects.
We first create two simple objects which have differences and similarities. They are strings which are system.objects.
@' Although the Borgias were rather gorgeous they liked the absurder kinds of murder Anon '@>"$env:Temp\Firstpoem.txt" @' Here lies John Bunn who was killed by a gun his name wasn't bun but Wood 'Wood' wouldn't rhyme with gun but 'Bunn' would Anon '@>"$env:Temp\Secondpoem.txt"
Now we compare these two objects
compare-object (Get-Content "$env:Temp\poem.txt")(Get-Content "$env:Temp\Secondpoem.txt") -IncludeEqual
InputObject SideIndicator
----------- -------------
==
Anon ==
Here lies John Bunn =>
who was killed by a gun =>
his name wasn't bun but Wood =>
'Wood' wouldn't rhyme with gun but 'Bunn' would =>
Although the Borgias <=
were rather gorgeous <=
they liked the absurder <=
kinds of murder <=
OK. What that means is that ‘Anon’ and the blank line are in both equal in both (==) , and the rest are either just in the First poem (<=) or else in the second (=>)
I’d much prefer a side-by-side comparison that you can filter to just show the differences
Ref Source Target Match --- ------ ------ ----- $[0] Although the Borgias Here lies John Bunn <> $[1] were rather gorgeous who was killed by a gun <> $[2] they liked the absurder his name wasn't bun but Wood <> $[3] kinds of murder 'Wood' wouldn't rhyme with gun but 'Bunn' would <> $[4] == $[5] Anon Anon ==
OK. Let’s give it something more complicated.
$TheResult= #our first list of employees
@'
{
"employees": [
{
"resigned": "false",
"salary": 275,
"lastName": "Doe",
"Enddate": null,
"warnings": [234, 678,3453, 67],
"firstName": "John"
},
{
"EndDate": "2012-04-23T18:25:43.511Z",
"resigned": false,
"firstName": "Anna",
"salary": 300,
"lastName": "Smith"
},
{
"EndDate": "2018-04-23T18:25:43.511Z",
"resigned": false,
"firstName": "Peter",
"salary": 400,
"lastName": "Jones"
}
]
}
'@ | convertFrom-json
$TheOtherResult= #our revised list of employees
@'
{
"employees": [
{
"resigned": "true",
"salary": 275,
"lastName": "Doe",
"Enddate": null,
"warnings": [234, 678,3453,56, 67],
"firstName": "John"
},
{
"EndDate": "2012-04-23T18:25:43.511Z",
"resigned": false,
"firstName": "Anna",
"salary": 350,
"lastName": "Smith"
},
{
"EndDate": "2018-04-23T18:25:43.511Z",
"resigned": false,
"firstName": "Peter",
"salary": 400,
"lastName": "Jones"
}
]
}
'@ | convertFrom-json
#So we ask PowerShell what the differences are
Compare-object -ReferenceObject $TheResult.employees `
-DifferenceObject $TheOtherResult.employees `
-IncludeEqual `
-Property @('salary','Firstname','Lastname','Enddate','warnings','Resigned')
Well. We get this…
salary : 400
Firstname : Peter
Lastname : Jones
Enddate : 2018-04-23T18:25:43.511Z
warnings :
Resigned : False
SideIndicator : ==
salary : 275
Firstname : John
Lastname : Doe
Enddate :
warnings : {234, 678, 3453, 56...}
Resigned : true
SideIndicator : =>
salary : 350
Firstname : Anna
Lastname : Smith
Enddate : 2012-04-23T18:25:43.511Z
warnings :
Resigned : False
SideIndicator : =>
salary : 275
Firstname : John
Lastname : Doe
Enddate :
warnings : {234, 678, 3453, 67}
Resigned : false
SideIndicator : <=
salary : 300
Firstname : Anna
Lastname : Smith
Enddate : 2012-04-23T18:25:43.511Z
warnings :
Resigned : False
SideIndicator : <=
Which tells us that two objects in the array are the same (==), two are only in the second object (=>) and two are in the first (<=). However, it doesn’t tell us that Anna’s salary is the difference. To get into the detail, I want something like this.
Ref Source Target Match --- ------ ------ ----- $.employees[0].firstName John John == $.employees[0].lastName Doe Doe == $.employees[0].resigned false true <> $.employees[0].salary 275 275 == $.employees[0].warnings[0] 234 234 == $.employees[0].warnings[1] 678 678 == $.employees[0].warnings[2] 3453 3453 == $.employees[0].warnings[3] 67 56 <> $.employees[0].warnings[4] 67 -> $.employees[1].EndDate 2012-04-23T18:25:43.511Z 2012-04-23T18:25:43.511Z == $.employees[1].firstName Anna Anna == $.employees[1].lastName Smith Smith == $.employees[1].resigned False False == $.employees[1].salary 300 350 <> $.employees[2].EndDate 2018-04-23T18:25:43.511Z 2018-04-23T18:25:43.511Z == $.employees[2].firstName Peter Peter == $.employees[2].lastName Jones Jones == $.employees[2].resigned False False ==
I want a ‘diff’ or difference of the entire object to see what values have changed. I’d quite like to specify the depth to compare, and what to exclude or include, especially with larger objects. You’ll notice that this result can be filtered to allow you to list just properties that are different or missing.
Problems with object comparisons.
Comparing arrays
Before we delve too far into how Diff-objects works, we need to mention a general problem with comparing arrays. Whereas, when values have unique keys it is easy to determine differences, There is a whole branch of computer science to work out how, without unique keys, you can compare two versions of an array in a way that shows how it has changed. One of the problems is that the insertion, rather the appending, of an array element makes every subsequent element different to the reference. In terms of text represented by arrays of lines, a carriage return makes everything after it a difference. Where the elements in an array are ordered, and the order is significant, it is legitimate to do this. Where they are random, and you allow duplicate values, then you must match them iteratively, one pair at a time, and then remove them as candidates for the subsequent matches.
In JSON, the order of arrays is significant, so I take this as a precedent to do the easy option.
Comparing nulls
NULLs have a variety of meanings. They can mean ‘Unknown’, which means that the result of comparing an unknown with something is always unknown. There again, a blank string is often used as if it were a NULL string. PowerShell can get confusing because it is very difficult to compare an object that doesn’t exist, and therefore returns NULL when you reference it, with an object that has a null value when you reference it. When comparing objects and their values, you need to know the difference.
I can’t find a consensus view on whether a blank string is the same as a null value. It isn’t in relational databases. One is known to be a blank string and the other is unknown. I’ve made it configurable. If you need to equate null and Blank, just set –NullAndBlankSame to $true. This is useful where you store objects as CSV because CSV has no consistent concept of a null string.
Avoiding stuff
Starting a comparison at some reference point is easy: you just specify the reference point where you start the comparison in the -Parent parameter. Ignoring embedded objects is trickier. A classical example is ignoring comments in XML files. (#comment). A far worst problem is presented by those monster arrays and god-like objects that you tend to find in .NET objects passed to you from the operating system. You can specify a list (array) of strings with the names of the objects or references (those strings in the first column) that you wish to avoid, such as ‘$.employees[2].resigned‘ in the last example.
How Diff-Objects works
I’ve done a blog post describing a Cmdlet I’ve called Display-Object. Although I’ve found it to be a very useful Cmdlet in its own right, I felt that it was a useful stepping-stone in understanding how I’ve tackled the problem of ‘diffing’ objects (finding the difference between them). I started writing Display-Object just as an illustration but, as so often in life, I got rather interested in it because it proved so useful to me in finding out what was going on inside some tricky objects.
Basically, a lot of object comparisons just ‘walk’ the hierarchy of a reference object, comparing any ‘comparable object’ (most simple values) with the same reference in the difference object This is like a left outer join. Because I want to see additions and subtractions I do the equivalent of a full outer join to find the differences
It doesn’t report what objects are different, just the values. If an object has differences in the values between the reference object and its equivalent in the difference object then you can be sure there is a difference. By ‘difference’, I include those records that appear only in one of the two objects.
Any useful Cmdlet that is designed to participate in a pipeline need to report the result of a comparison via a collection of psCustomObjects. In this way, one can use Select-Object, Where-Object and all those other useful participants. Although it introduces some redundancies, I use a four-column format.
- The first column is the path expression to the object. By this I mean the dot references and array indices. A dollar sign means the name of the object, as with most object paths. Basically, in PowerShell in the ISE, you add the reference except for the dollar sign to the variable referring to the object, execute it, and you’ll see the value.
- The second column is the value in the reference object
- The Third column is the value in the difference object.
- The fourth column contains a symbol that gives the result of the comparison. This can be
- ‘Both there and equal’ (‘==’)
- ‘Both there and different’ (‘<>’)
- ‘Only in the difference object’ (‘->’)
- ‘Only in the reference object’ (‘<-’)
- ‘Could not be compared (e.g. write-only values) (‘–’)
Uses for an ‘Object Diff’
Checking test results
Most Cmdlets produce objects. These are often lists of PS Custom Objects. If you can look at the output in Format-Table, that’s probably the case. Unless they are huge, these objects can be rendered as a document that can be saved. The ConvertTo … series of cmdlets are good for this. If the data is essentially tabular you can save it in its most economical form as CSV, but JSON is OK. This often gives you the opportunity to test your cmdlets as you develop them. You work out, and get general agreement about, what the result should look like for a particular set of parameters. If your cmdlet produces the same result for the same parameters, then you have a degree of confidence that you haven’t broken anything.
Normally, you’d want to keep your test materials on-disk and iterate through them. Just to illustrate how it works, though, I’ve done it in code. I’ve create a file-based dataset that represents what the Display-object Cmdlet actually should be producing, together with the object that we’re displaying. We want to make sure that Display-object still works after we alter it.
#A Test for a Display-Object Cmdlet that we are developing.
#We have the reference version of what the data should be in #ref
$Ref=@'
#TYPE System.Management.Automation.PSCustomObject
"Path","Value"
"$.Ham.Downtime",
"$.Ham.Location","Floor two rack"
"$.Ham.Users[0]","Fred"
"$.Ham.Users[1]","Jane"
"$.Ham.Users[2]","Mo"
"$.Ham.Users[3]","Phil"
"$.Ham.Users[4]","Tony"
"$.Ham.version","2019"
"$.Japeth.Location","basement rack"
"$.Japeth.Users[0]","Karen"
"$.Japeth.Users[1]","Wyonna"
"$.Japeth.Users[2]","Henry"
"$.Japeth.version","2008"
"$.Shem.Location","Server room"
"$.Shem.Users[0]","Fred"
"$.Shem.Users[1]","Jane"
"$.Shem.Users[2]","Mo"
"$.Shem.version","2017"
'@ |ConvertFrom-Csv
# We now have the reference result. we now create the test input
$ServersAndUsers =
@{'Shem' =
@{
'version' = '2017'; 'Location' = 'Server room';
'Users'=@('Fred','Jane','Mo')
};
'Ham' =
@{
'version' = '2019'; 'Location' = 'Floor two rack';
'Downtime'=$null
'Users'=@('Fred','Jane','Mo','Phil','Tony')
};
'Japeth' =
@{
'version' = '2008'; 'Location' = 'basement rack';
'Users'=@('Karen','Wyonna','Henry')
}
}
#we run the 'Display-Object' that we are developing.
$Diff= Display-Object $ServersAndUsers
# we now have a #Ref object with what the output should be, and we have the $diff object
# of what is produced by the current version
# We test to see if the $Ref and $Diff match.
$TestResult=Diff-Objects -Ref $ref -Diff $diff -NullAndBlankSame $True |
where {$_.Match -ne '=='}
if ($TestResult) #if any differences were reported.
{Write-warning 'Test for Display-Object with ServersAndUsers failed'
$TestResult|format-table}
You can run this, but instead of changing the code in Diff-object, we can take the easier route and simply change the data and seeing if this is picked up by tester.
Seeing how data in large objects change.
The important point here with a large object is to only look at what you are interested in. Even a conceptually-simple object like a data table can end up with a lot of nooks and crannies full of data. A process object can be severely over-weight. You can start by just surveying the branch you’re interested in by specifying the ‘dot’ address of the data that you need to see, and avoid all the data-carbohydrates. Here, in the first example, we are checking the process where you aren’t at all interested in those arrays, so you filter them out by listing them in the ‘avoid’ parameter..
$process=(get-process pwsh)
#<some time later>
Diff-Objects $process (get-process pwsh) -Depth 3 -Avoid @('Modules','Threads','StartInfo') -NullAndBlankSame $true
You can start ‘some way from the trunk’ by presenting the cmdlet with the same reference, and providing the parentage to the ‘parent’ parameter so that the reference is correct if you subsequently want to get an individual value. Notice that we’ve not only carved off the branch of the data we’re interested in but we’ve specified this address as ‘parent’ so that the address is correct too.
Diff-Objects $process.MainModule (get-process pwsh).MainModule -Depth 3 -Parent '$.MainModule' -NullAndBlankSame $true
The Code
The code to for this utility is on Github. It is a bit bulky, and I feel that the code will change over time, so it might be best to get it from Github, as it is always trickier to update a published article.
<#
.SYNOPSIS
Used to Compare two powershell objects
.DESCRIPTION
This compares two powershell objects by determining their shared
keys or array sizes and comparing the values of each. It uses the
.PARAMETER Ref
The source object
.PARAMETER diff
The target object
.PARAMETER Avoid
a list of any object you wish to avoid comparing
.PARAMETER Parent
Only used for recursion
.PARAMETER Depth
The depth to which you wish to recurse
.PARAMETER CurrentDepth
Only used for recursion
.PARAMETER NullAndBlankSame
Do we regard null and Blank the same for the purpose of comparisons.
.NOTES
Additional information about the function.
#>
function Diff-Objects
{
param
(
[Parameter(Mandatory = $true,
Position = 1)]
[object]$Ref,
[Parameter(Mandatory = $true,
Position = 2)]
[object]$Diff,
[Parameter(Mandatory = $false,
Position = 3)]
[object[]]$Avoid = @('Metadata', '#comment'),
[Parameter(Mandatory = $false,
Position = 4)]
[string]$Parent = '$',
[Parameter(Mandatory = $false,
Position = 5)]
[int]$Depth = 4,
[Parameter(Mandatory = $false,
Position = 6)]
[int]$CurrentDepth = 0,
[Parameter(Position = 7)]
[boolean]$NullAndBlankSame = $False
)
if ($CurrentDepth -eq $Depth) { Return };
# first create a unique (unduplicated) list of all the key names obtained from
# either the source or target object
$SourceInputType = $Ref.GetType().Name
$TargetInputType = $Diff.GetType().Name
if ($SourceInputType -in 'HashTable', 'OrderedDictionary')
{
$Ref = [pscustomObject]$Ref;
$SourceInputType = 'PSCustomObject'
}
if ($TargetInputType -in 'HashTable', 'OrderedDictionary')
{
$Diff = [pscustomObject]$Diff;
$TargetInputType = 'PSCustomObject'
}
$InputType = $SourceInputType #we discard different types as different!
#are they both value types?
if ($Ref.GetType().IsValueType -and $Diff.GetType().IsValueType)
{
$Nodes = [pscustomobject]@{ 'Name' = ''; 'Match' = ''; 'SourceValue' = $Ref; 'TargetValue' = $Diff; }
}
elseif ($sourceInputType -ne $TargetInputType)
{
$Nodes = [pscustomobject]@{ 'Name' = ''; 'Match' = '<>'; 'SourceValue' = $Ref; 'TargetValue' = $Diff; }
}
elseif ($InputType -eq 'Object[]') # is it an array?
{
#iterate through it to get the array elements from both arrays
$ValueCount = if ($Ref.Count -ge $Diff.Count)
{ $Ref.Count }
else { $Diff.Count }
$Nodes = @{ }
$Nodes =
@(0..($ValueCount - 1)) | foreach{
$TheMatch = ''
if ($_ -ge $ref.count) { $TheMatch = '->' }
if ($_ -ge $Diff.count) { $TheMatch = '<-' }
$_ | Select @{ Name = 'Name'; Expression = { "[$_]" } },
@{ Name = 'Match'; Expression = { $TheMatch } },
@{ Name = 'SourceValue'; Expression = { $Ref[$_] } },
@{ Name = 'TargetValue'; Expression = { $Diff[$_] } }
}
}
#process the name/value objects
else
{
if ($InputType -in @('Hashtable', 'PSCustomObject'))
{
[string[]]$RefNames = [pscustomobject]$Ref | gm -MemberType NoteProperty | foreach{ $_.Name };
[string[]]$DiffNames = [pscustomobject]$Diff | gm -MemberType NoteProperty | foreach{ $_.Name };
}
else
{
[string[]]$RefNames = $Ref | gm -MemberType Property | foreach{ $_.Name };
[string[]]$DiffNames = $Diff | gm -MemberType Property | foreach{ $_.Name };
}
#the nodes can all be obtained by dot references
$Nodes = $RefNames + $DiffNames | select -Unique | foreach{
#Simple values just won't go down the pipeline, just keynames
# see if the key is there and if so what type of value it has
$Name = $_;
$index = $null;
$Type = $Null; #because we don't know it and it may not exist
$SourceValue = $null; #we fill this where possible
$TargetValue = $null; #we fill this where possible
if (($Name -notin $Avoid) -and ($Parent -notin $Avoid))
#if the user han't asked for it to be avoided
{
try
{
$TheMatch = $null;
if ($Name -notin $DiffNames) #if it isn't in the target
{
$TheMatch = '<-' #meaning only in the source
$SourceValue = $Ref.($Name)
#logically the source has a value but it may be null
}
elseif ($Name -notin $RefNames) #if it isn't in the source
{
$TheMatch = '->' #meaning only in the target
$TargetValue = $Diff.($Name)
# and logically the target has a value, perhaps null
}
else # it is OK to read both
{
$TargetValue = $Diff.($Name);
$SourceValue = $Ref.($Name)
if ($Null -eq $TargetValue -or $Null -eq $SourceValue)
{
write-Verbose '...one is a null'
if ($NullAndBlankSame -and
[string]::IsNullOrEmpty($TargetValue) -and
[string]::IsNullOrEmpty($SourceValue))
{ $TheMatch = '==' }
else
{
$TheMatch = "$(if ($Null -eq $Ref) { '-' }
else { '<' })$(if ($Null -eq $Diff) { '-' }
else { '>' })"
}
}
}
}
catch
{ $TargetValue = $null; $SourceValue = $null; $TheMatch = '--' }
$_ | Select @{ Name = 'Name'; Expression = { ".$Name" } },
@{ Name = 'Match'; Expression = { $TheMatch } },
@{ Name = 'SourceValue'; Expression = { $SourceValue } },
@{ Name = 'TargetValue'; Expression = { $TargetValue } }
}
}
}
$Nodes | foreach{
#Write-verbose $_| Format-Table
$DisplayableTypes = @('string', 'byte', 'boolean', 'decimal', 'double',
'float', 'single', 'int', 'int32', 'int16', 'intptr', 'long',
'int64', 'sbyte', 'uint16', 'null', 'uint32', 'uint64')
$DisplayableBaseTypes = @('System.ValueType', 'System.Enum')
$DiffType = 'NULL'; $DiffBaseType = 'NULL'; $RefType = 'NULL'; $RefBaseType = 'NULL';
$ItsAnObject = $null; $ItsAnArray = $null; $ItsAComparableValue = $Null;
$name = $_.Name; $TheMatch = $_.Match;
$SourceValue = $_.SourceValue; $TargetValue = $_.TargetValue;
$FullName = "$Parent$inputName$Name";
# now find out its type
if ($_.SourceValue -ne $null)
{
$RefType = $_.SourceValue.GetType().Name;
$RefBaseType = $_.SourceValue.GetType().BaseType
$RefDisplayable = (($RefType -in $DisplayableTypes) -or ($RefBaseType -in $DisplayableBaseTypes))
}
if ($_.TargetValue -ne $null)
{
$DiffType = $_.TargetValue.GetType().Name;
$DiffBaseType = $_.TargetValue.GetType().BaseType
$DiffDisplayable = (($DiffType -in $DisplayableTypes) -or ($DiffBaseType -in $DisplayableBaseTypes))
}
$ItsAComparableValue = $false; # until proven otherwise
if ($TheMatch -eq $null -or $TheMatch -eq '') # if no match done yet
{
If ($RefDisplayable -and $DiffDisplayable)
{
#just compare the values
if ($SourceValue -eq $TargetValue)
{ $TheMatch = '==' } # the same
else { $TheMatch = '<>' } # different
$ItsAComparableValue = $true;
}
}
#is it an Array?
$ItsAnArray = $RefType -in 'Object[]';
#is it an object?
$ItsAnObject = ($RefBaseType -in @(
'System.Xml.XmlLinkedNode', 'System.Xml.XmlNode',
'System.Object', 'System.ComponentModel.Component'
))
if (!($TheMatch -eq $null -or $TheMatch -eq ''))
{
#if we have a match
if ($ItsAnObject)
{
$TheTypeItIs = '(Object)';
} #as a display reference
if ($ItsAnArray)
{
$TheTypeItIs = '[Array]'; $FullName = "$Parent$Name" #as a display reference
};
#create a sensible display for object values
$DisplayedValue = @($SourceValue, $TargetValue) | foreach{
if ($ItsAComparableValue) { $_ }
elseif ($_ -ne $Null) { $_.GetType().Name }
else { '' }
}
if ($RefDisplayable -and ($DisplayedValue)) { $DisplayedValue[0] = $SourceValue }
if ($DiffDisplayable -and ($DisplayedValue)) { $DisplayedValue[1] = $TargetValue }
# create the next row of our 'table' with a pscustomobject
1 | Select @{ Name = 'Ref'; Expression = { $FullName } },
@{ Name = 'Source'; Expression = { $DisplayedValue[0] } },
@{ Name = 'Target'; Expression = { $DisplayedValue[1] } },
@{ Name = 'Match'; Expression = { $TheMatch } }
}
else
{
if (($ItsAnObject) -or ($ItsAnArray))
{
#if it is an object or array on both sides
Diff-Objects $SourceValue $targetValue $Avoid "$Fullname" $Depth ($CurrentDepth + 1) $NullAndBlankSame
}
# call the routine recursively
else { write-warning "No idea what to do with object of named '$($Name)', basetype '$($RefBaseType)''$($DiffBaseType)' with match of '$TheMatch'" }
if ($RefType -ne $Null)
{ write-Verbose "compared [$RefType]$FullName $RefDisplayable $DiffDisplayable '($RefBaseType)'-- '($DiffBaseType)' with '$TheMatch' match" }
}
}
}
Conclusion
Almost every PowerShell task that involves comparing objects seems to come up with another requirement. The built-in Compare-Objects cmdlet is a good start and can be persuaded to do a lot of tasks, but nothing beats a cmdlet with the source that one can alter to suit. I don’t like to develop anything I don’t immediately need for my work so I’m happy to leave something that does the job. One day I may come up with, for example with the need for a cmdlet that lists EVERY key/value pair in the target that doesn’t appear in the source, rather than to stop at the level of the first difference. I may need something that will compares arrays where the order is not significant. No worries, because now I can just add it and test it!
The post Diff-Objects: a PowerShell utility Cmdlet for Discovering Object differences appeared first on Simple Talk.
from Simple Talk https://ift.tt/3zLDqB4
via
No comments:
Post a Comment