I tried to test in parallel. But from log, I could know that Xcode on GitHub Action only use one simulator and just messing up the ordering of the tests which makes hard to reproduce test fail.
First of all, I should use xcpretty. If not, because the log is too long just the build fails. And even if some tests fails, the log that shown on GitHub only shows which tests failed without reasons.
How to view log of test fail on Travis
Debug build
xcodebuild (because the logs get deleted)
After test finished, cd /Users/travis/Library/Developer/Xcode/DerivedData/[Your Project]-[some code- shown during the process or log]/Logs/Test/Test-[Your Project]-2019.04.11_06-20-41-+0000.xcresult/1_Test/Diagnostics/
You can drill down to [Product_name]Tests folders
and vim StandardOutputAndStandardError.txt or vim Session-[Product_name]-[date]~~.log
When test failed only on Travis
increase waiting time
check whether a variable is kept used between tests affecting other test
When you want to have multiple parameters to the command & skip certain file testing, use skip-testing. It’s useful when you skip testing network dependent tests on CI.
When write -skip-testing “File”. Just add the file path under your project. Does not include “group” in project navigator.
And “iPhone 6” means iPhone 6. It’s not code like iPhone8,2
If you get error like below, try deleting “CODE_SIGNING_REQUIRED=NO”
â–¸ Signing /Users/travis/Library/Developer/Xcode/DerivedData/Something-gfrkrbrfwqplztgzqocbzyotbmpg/Build/Products/Debug-iphonesimulator/SomethingUITests-Runner.app/PlugIns/SomethingUITests.xctest
2018-08-10 05:51:50.674 xcodebuild[8511:16979] Error Domain=IDETestOperationsObserverErrorDomain Code=6 "Early unexpected exit, operation never finished bootstrapping - no restart will be attempted" UserInfo={NSLocalizedDescription=Early unexpected exit, operation never finished bootstrapping - no restart will be attempted}
But If you use this script in travis, don’t forget to set pipe fail. If you do not use pipe fail, xcpretty’s result will be the final result of travis showing “pass” everytime.
pipefail: If set, the return value of a pipeline is the value of the last (rightmost)command to exit with a non-zero status, or zero if all commands in the pipelineexit successfully. This option is disabled by default.
set -o pipefail && xcodebuild -workspace Something.xcworkspace -scheme Something -sdk iphonesimulator11.4 -destination ‘platform=iOS Simulator,name=iPhone 8,OS=11.4’ -skip-testing:SomethingUITests -skip-testing:SomethingTests/ModelInitWithServerTests test | xcpretty
If xcodebuild does not execute simulator, just run the same simulator before running script.