Optionally keep paths that do not exist with `resolve` #11

New issue

Closed

opened 2025-10-11 09:41:42 +00:00 by UserCurt · 3 comments

UserCurt commented

2025-10-11 09:41:42 +00:00

(Migrated from codeberg.org)

This feature request is to optionally allow mpath.resolve to return MPath Pointers that do not point to existing entries (i.e., the paths do not exist).

The proposed solution is to replace the named argument AllowEmptyFields with a new one NullField which dictates behavior when a null field is encountered. When a field that does not exist is encountered, the behavior will be dictated by the value of Nullfield:

omit -- Ignores the invalid/null path and excludes it from the result
missing -- Replaces the invalid/null path with a missing Pointer
retain -- Keeps the invalid/null path
error -- Gives an error

There is some overlap in functionality between this proposal and FillGaps with some nuanced differences when working with heterogeneous data structures. These details may not matter to the end user, so merging FillGaps with NullField should be considered.

This feature request is to optionally allow `mpath.resolve` to return MPath Pointers that do not point to existing entries (i.e., the paths do not exist). The proposed solution is to replace the named argument `AllowEmptyFields` with a new one `NullField` which dictates behavior when a null field is encountered. When a field that does not exist is encountered, the behavior will be dictated by the value of `Nullfield`: - `omit` -- Ignores the invalid/null path and excludes it from the result - `missing` -- Replaces the invalid/null path with a missing Pointer - `retain` -- Keeps the invalid/null path - `error` -- Gives an error There is some overlap in functionality between this proposal and `FillGaps` with some nuanced differences when working with heterogeneous data structures. These details may not matter to the end user, so merging `FillGaps` with `NullField` should be considered.

UserCurt commented

2025-10-11 09:54:42 +00:00

(Migrated from codeberg.org)

FillGaps might need to be generalized to include missing fields:

S = struct(A=struct(X=1, Y=2), B=struct(Y=3, Z=4));
mpath.resolve(S, "/*/*")
% returns:
%   [/A/X,      /A/Y, <missing>;
%    <missing>, /B/Y, /B/Z     ]

The open question is what option (named argument) should be provided so that the user can get this instead:

%   [/A/X, /A/Y, /A/Z;
%    /B/X, /B/Y, /B/Z ]

A related question: Should /{A,B}/{X,Y,Z} (once set notation from #3 is implemented) behave identically to /*/* in this circumstance? I can see the argument how the fields X, Y, and Z must be checked for each branch A and B in the former, but for the later each * at each branch is considered independent and can match different fields. However, everything eventually has to be shaped anyways, so the second * effectively share the same pool of fields, suggesting these should behave the same. This line of reasoning suggests FillGaps and NullField should be merged somehow. That is, unless there is a use case otherwise?

`FillGaps` might need to be generalized to include missing fields: ```matlab S = struct(A=struct(X=1, Y=2), B=struct(Y=3, Z=4)); mpath.resolve(S, "/*/*") % returns: % [/A/X, /A/Y, <missing>; % <missing>, /B/Y, /B/Z ] ``` The open question is what option (named argument) should be provided so that the user can get this instead: ```matlab % [/A/X, /A/Y, /A/Z; % /B/X, /B/Y, /B/Z ] ``` A related question: Should `/{A,B}/{X,Y,Z}` (once set notation from #3 is implemented) behave identically to `/*/*` in this circumstance? I can see the argument how the fields X, Y, and Z must be checked for each branch A and B in the former, but for the later each * at each branch is considered independent and can match different fields. However, everything eventually has to be shaped anyways, so the second `*` effectively share the same pool of fields, suggesting these should behave the same. This line of reasoning suggests `FillGaps` and `NullField` should be merged somehow. That is, unless there is a use case otherwise?

UserCurt commented

2026-06-25 02:03:14 +00:00

(Migrated from codeberg.org)

Some progress has been made on this issue but it is not yet complete.

Two notes:

Having resolve be able to return paths that do not exist will help enable set to create fields that do not exist.
FillGaps was historically overloaded in get to handle two distinct situations: Filling in gaps where paths were missing (via missing paths returned from resolve) and filling in the gaps when there was a size mismatch. It was later split into the now MissingPolicy and SizeMismatchPolicy.

Relevant history around resolve:

137593e on 2025-09-25: AllowEmptyMatches and FillGaps originally introduced using logical policy.
5df9c181 on 2025-10-10: This was changed where AllowEmptyMatches was replaced with NullField in resolve with more options, now supporting for "omit" "missing", "retain", "error".
70f5380 on 2025-10-16: Prior to this point, NullField determined what to do when a missing path branch was found, while FillGaps determined whether post-reshape rectangular holes were allowed. In this commit, NullField was renamed (and old AllowEmptyMatches language) to MissingPolicy in resolve, and FillGaps was removed from resolve.

The new MissingPolicy wasn't fully functional at this time but was addressed in later commits:

bd5ddcd: retained path reconstruction for missing resolved slots
df3e54f: heterogeneous struct-array missing-policy bug
65af1e0: missing branch handling
84f9e78: DotSet with MissingPolicy="retain" including missing fields

The new MissingPolicy approach was then used with get and other functions:

90a851f on 2026-06-11: While some changes were being underwent with resolve, get was disabled. This commit re-enabled get.
70ff089 on 2026-06-13: Policy semantics in get became better controlled by having MissingPolicy for handling missing paths and SizeMismatchPolicy for handling mismatched sizes in the results.
206e68a and 83e1046 on 2026-06-13: set and remove were modified to use MissingPolicy but with different options based on what was reasonable for those functions: set accepted "create", "ignore", and "error", and remove accepted "ignore" and "error".

This issue is not fully addressed yet. Below is an example that does not return results as expected:

>> data = struct(Y = struct(A = 1, B = 2));
>> mp = mpath.Pointer([mpath.operation.DotSet(["X", "Y"]) mpath.operation.DotGlob("*")])

mp = 

  mpath: /{X,Y}/*

>> mp.resolve(data, MissingPolicy='omit')

ans = 

  1×2 mpath array

    /Y/A    /Y/B

>> mp.resolve(data, MissingPolicy='missing')

ans = 

  2×3 mpath array

    <missing>    <missing>    <missing>
    <missing>    /Y/A         /Y/B

>> mp.resolve(data, MissingPolicy='retain')

ans = 

  2×3 mpath array

    /X/*    /X/A    /X/B
    /Y/*    /Y/A    /Y/B

The expected results would be:

>> mp.resolve(data, MissingPolicy='omit')

ans = 

  1×2 mpath array

    /Y/A    /Y/B

>> mp.resolve(data, MissingPolicy='missing')

ans = 

  2×3 mpath array

    <missing>    <missing>
    /Y/A         /Y/B

>> mp.resolve(data, MissingPolicy='retain')

ans = 

  2×3 mpath array

    /X/A    /X/B
    /Y/A    /Y/B

Some progress has been made on this issue but it is not yet complete. Two notes: - Having `resolve` be able to return paths that do not exist will help enable `set` to create fields that do not exist. - `FillGaps` was historically overloaded in `get` to handle two distinct situations: Filling in gaps where paths were missing (via missing paths returned from `resolve`) and filling in the gaps when there was a size mismatch. It was later split into the now `MissingPolicy` and `SizeMismatchPolicy`. Relevant history around `resolve`: - `137593e` on `2025-09-25`: `AllowEmptyMatches` and `FillGaps` originally introduced using logical policy. - `5df9c181` on `2025-10-10`: This was changed where `AllowEmptyMatches` was replaced with `NullField` in `resolve` with more options, now supporting for `"omit"` `"missing"`, `"retain"`, `"error"`. - `70f5380` on `2025-10-16`: Prior to this point, `NullField` determined what to do when a missing path branch was found, while `FillGaps` determined whether post-reshape rectangular holes were allowed. In this commit, `NullField` was renamed (and old `AllowEmptyMatches` language) to `MissingPolicy` in `resolve`, and `FillGaps` was removed from `resolve`. The new `MissingPolicy` wasn't fully functional at this time but was addressed in later commits: - `bd5ddcd`: retained path reconstruction for missing resolved slots - `df3e54f`: heterogeneous struct-array missing-policy bug - `65af1e0`: missing branch handling - `84f9e78`: `DotSet` with `MissingPolicy="retain"` including missing fields The new `MissingPolicy` approach was then used with `get` and other functions: - `90a851f` on `2026-06-11`: While some changes were being underwent with `resolve`, `get` was disabled. This commit re-enabled `get`. - `70ff089` on `2026-06-13`: Policy semantics in `get` became better controlled by having `MissingPolicy` for handling missing paths and `SizeMismatchPolicy` for handling mismatched sizes in the results. - `206e68a` and `83e1046` on `2026-06-13`: `set` and `remove` were modified to use `MissingPolicy` but with different options based on what was reasonable for those functions: `set` accepted `"create"`, `"ignore"`, and `"error"`, and `remove` accepted `"ignore"` and `"error"`. --- This issue is not fully addressed yet. Below is an example that does not return results as expected: ```matlab >> data = struct(Y = struct(A = 1, B = 2)); >> mp = mpath.Pointer([mpath.operation.DotSet(["X", "Y"]) mpath.operation.DotGlob("*")]) mp = mpath: /{X,Y}/* >> mp.resolve(data, MissingPolicy='omit') ans = 1×2 mpath array /Y/A /Y/B >> mp.resolve(data, MissingPolicy='missing') ans = 2×3 mpath array <missing> <missing> <missing> <missing> /Y/A /Y/B >> mp.resolve(data, MissingPolicy='retain') ans = 2×3 mpath array /X/* /X/A /X/B /Y/* /Y/A /Y/B ``` The expected results would be: ``` >> mp.resolve(data, MissingPolicy='omit') ans = 1×2 mpath array /Y/A /Y/B >> mp.resolve(data, MissingPolicy='missing') ans = 2×3 mpath array <missing> <missing> /Y/A /Y/B >> mp.resolve(data, MissingPolicy='retain') ans = 2×3 mpath array /X/A /X/B /Y/A /Y/B ```

UserCurt commented

2026-07-02 09:07:05 +00:00

(Migrated from codeberg.org)

The issue has been addressed in fdd36bb.

Current behavior from the example in the comment above gives the expected result shapes:

MissingPolicy="omit" -> 1×2
MissingPolicy="missing" -> 2×2
MissingPolicy="retain" -> 2×2

Related but more specific bugs are being tracked separately:

#32 for paths that continue through non-container values
#31 for behavior with empty resolved arrays

The issue has been addressed in `fdd36bb`. Current behavior from the example [in the comment above](https://codeberg.org/UserCurt/mpath/issues/11#issuecomment-18031568) gives the expected result shapes: - `MissingPolicy="omit"` -> `1×2` - `MissingPolicy="missing"` -> `2×2` - `MissingPolicy="retain"` -> `2×2` Related but more specific bugs are being tracked separately: - #32 for paths that continue through non-container values - #31 for behavior with empty resolved arrays