[tests] Improve the CustomizedCodeSigning tests to not depend on file order. #20889

rolfbjarne · 2024-07-12T11:51:12Z

The CustomizedCodeSigning test asserts that a certain condition shows a
particular error message from codesign.

There are multiple files in the app bundle that can trigger this particular
message, so change the logic to not assert on a particular file, instead use
assert on the remained of the error message.

Fixes this random test failure:

Xamarin.Tests.DotNetProjectTest.CustomizedCodeSigning(iOS,"ios-arm64"): Failure when comparing error messages:
    Unexpected error message #0:
        Expected: /usr/bin/codesign exited with code 1:\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all\nIn subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Diagnostics.DiagnosticSource.dll
        Actual: /usr/bin/codesign exited with code 1:\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all\nIn subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Collections.NonGeneric.aotdata.arm64
    Unexpected error message #1:
        Expected: Failed to codesign '/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app': /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all\nIn subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Diagnostics.DiagnosticSource.dll
        Actual: Failed to codesign '/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app': /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all\nIn subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Collections.NonGeneric.aotdata.arm64
    All errors:
        /usr/bin/codesign exited with code 1:
            /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature
            /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all
            In subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Collections.NonGeneric.aotdata.arm64
                    Failed to codesign '/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app': /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature
            /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all
            In subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Collections.NonGeneric.aotdata.arm64

… order. The CustomizedCodeSigning test asserts that a certain condition shows a particular error message from codesign. There are multiple files in the app bundle that can trigger this particular message, so change the logic to not assert on a particular file, instead use assert on the remained of the error message. Fixes this random test failure: Xamarin.Tests.DotNetProjectTest.CustomizedCodeSigning(iOS,"ios-arm64"): Failure when comparing error messages: Unexpected error message #0: Expected: /usr/bin/codesign exited with code 1:\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all\nIn subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Diagnostics.DiagnosticSource.dll Actual: /usr/bin/codesign exited with code 1:\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all\nIn subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Collections.NonGeneric.aotdata.arm64 Unexpected error message #1: Expected: Failed to codesign '/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app': /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all\nIn subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Diagnostics.DiagnosticSource.dll Actual: Failed to codesign '/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app': /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature\n/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all\nIn subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Collections.NonGeneric.aotdata.arm64 All errors: /usr/bin/codesign exited with code 1: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all In subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Collections.NonGeneric.aotdata.arm64 Failed to codesign '/Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app': /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: replacing existing signature /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app: code object is not signed at all In subcomponent: /Users/builder/azdo/_work/4/s/xamarin-macios/tests/dotnet/CustomizedCodeSigning/iOS/bin/Debug/net9.0-ios/ios-arm64/CustomizedCodeSigning.app/System.Collections.NonGeneric.aotdata.arm64

…mparison

vs-mobiletools-engineering-service2 · 2024-08-08T13:39:40Z

📚 [CI Build] Artifacts 📚

Packages generated

View packages

Pipeline on Agent
Hash: [PR build]

vs-mobiletools-engineering-service2 · 2024-08-08T13:43:50Z

💻 [CI Build] Tests on macOS X64 - Mac Sonoma (14) passed 💻

✅ All tests on macOS X64 - Mac Sonoma (14) passed.

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-08T13:48:56Z

💻 [CI Build] Tests on macOS M1 - Mac Big Sur (11) passed 💻

✅ All tests on macOS M1 - Mac Big Sur (11) passed.

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-08T13:49:39Z

💻 [CI Build] Tests on macOS M1 - Mac Monterey (12) passed 💻

✅ All tests on macOS M1 - Mac Monterey (12) passed.

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-08T14:07:51Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-08T14:15:16Z

❌ [CI Build] Tests on macOS M1 - Mac Ventura (13) failed ❌

Failed tests are:

xammac_tests
monotouch-test

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-08T15:46:20Z

💻 [CI Build] Tests on macOS M1 - Mac Ventura (13) passed 💻

✅ All tests on macOS M1 - Mac Ventura (13) passed.

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-08T16:01:39Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-08T16:13:33Z

🚀 [CI Build] Test results 🚀

Test results

✅ All tests passed on VSTS: test results.

🎉 All 168 tests passed 🎉

Tests counts

✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (iOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 6 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 7 tests passed. Html Report (VSDrops) Download
✅ generator: All 2 tests passed. Html Report (VSDrops) Download
✅ install-source: All 1 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 7 tests passed. Html Report (VSDrops) Download
✅ introspection: All 8 tests passed. Html Report (VSDrops) Download
✅ linker: All 65 tests passed. Html Report (VSDrops) Download
✅ mac-binding-project: All 1 tests passed. Html Report (VSDrops) Download
✅ mmp: All 2 tests passed. Html Report (VSDrops) Download
✅ mononative: All 6 tests passed. Html Report (VSDrops) Download
✅ monotouch (iOS): All 11 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 7 tests passed. [attempt 2] Html Report (VSDrops) Download
✅ monotouch (macOS): All 8 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 11 tests passed. Html Report (VSDrops) Download
✅ monotouch (watchOS): All 4 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ mtouch: All 1 tests passed. Html Report (VSDrops) Download
✅ xammac: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 8 tests passed. Html Report (VSDrops) Download
✅ xtro: All 2 tests passed. Html Report (VSDrops) Download

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-09T10:57:48Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-09T11:58:07Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-09T17:13:52Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-09T19:29:44Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-12T10:25:25Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-12T11:53:18Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-12T12:24:30Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-12T13:00:57Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-12T13:37:42Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-12T14:09:31Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-12T14:41:06Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-12T15:13:09Z

❌ [CI Build] Windows Integration Tests failed ❌

❌ Failed ❌

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

vs-mobiletools-engineering-service2 · 2024-08-12T16:30:54Z

💻 [CI Build] Windows Integration Tests passed 💻

✅ All Windows Integration Tests passed.

Pipeline on Agent
Hash: 9d26a952d38eece46230a2621ff3811937dae10a [PR build]

rolfbjarne mentioned this pull request Jul 12, 2024

[net9.0] Merge main into net9.0. #20885

Merged

dalexsoto approved these changes Jul 12, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

haritha-mohan approved these changes Jul 16, 2024

View reviewed changes

Merge branch 'main' into dev/rolf/tests-fix-comparison

82e4164

This comment has been minimized.

Sign in to view

Merge branch 'main' into dev/rolf/tests-fix-comparison

73e0dfe

mandel-macaque approved these changes Aug 6, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

rolfbjarne added 2 commits August 8, 2024 13:28

Add some helper functions that are already in the net9.0 branch.

72a8edc

Merge remote-tracking branch 'origin/main' into dev/rolf/tests-fix-co…

9d26a95

…mparison

This comment has been minimized.

Sign in to view

rolfbjarne merged commit 7da0418 into main Aug 12, 2024
28 checks passed

rolfbjarne deleted the dev/rolf/tests-fix-comparison branch August 12, 2024 16:58

[tests] Improve the CustomizedCodeSigning tests to not depend on file order. #20889

[tests] Improve the CustomizedCodeSigning tests to not depend on file order. #20889

Conversation

rolfbjarne commented Jul 12, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

vs-mobiletools-engineering-service2 commented Aug 8, 2024

📚 [CI Build] Artifacts 📚

Packages generated

vs-mobiletools-engineering-service2 commented Aug 8, 2024

💻 [CI Build] Tests on macOS X64 - Mac Sonoma (14) passed 💻

vs-mobiletools-engineering-service2 commented Aug 8, 2024

💻 [CI Build] Tests on macOS M1 - Mac Big Sur (11) passed 💻

vs-mobiletools-engineering-service2 commented Aug 8, 2024

💻 [CI Build] Tests on macOS M1 - Mac Monterey (12) passed 💻

vs-mobiletools-engineering-service2 commented Aug 8, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 8, 2024

❌ [CI Build] Tests on macOS M1 - Mac Ventura (13) failed ❌

This comment has been minimized.

vs-mobiletools-engineering-service2 commented Aug 8, 2024

💻 [CI Build] Tests on macOS M1 - Mac Ventura (13) passed 💻

vs-mobiletools-engineering-service2 commented Aug 8, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 8, 2024

🚀 [CI Build] Test results 🚀

Test results

Tests counts

vs-mobiletools-engineering-service2 commented Aug 9, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 9, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 9, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 9, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 12, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 12, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 12, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 12, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 12, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 12, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 12, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 12, 2024

❌ [CI Build] Windows Integration Tests failed ❌

vs-mobiletools-engineering-service2 commented Aug 12, 2024

💻 [CI Build] Windows Integration Tests passed 💻