How to Run 5000+ Tests on Mobile Devices Everyday; From inDrive's Playbook (Part 2)

Posted by Ivan Grigoriev.

In the first part of this article, we described how to quickly and easily build an infrastructure for running UI tests on Android using Appium and Selenoid. We are continuing our story to explain how we incorporated the launch of UI tests in iOS into the process.

Scaling with GGR

The maximum number of parallel workflows within a host is limited by its resources. Thus, we needed a tool for combining multiple hosts into one cluster. For this, we use the Go Grid Router (GGR) from the folks at Aerokube. Based on the description in the documentation, GGR is a load balancer used for creating scalable and highly available Selenium clusters.

The project with the tests runs a query in GGR as part of the process used. It polls the Selenoid parameters specified in its configuration and distributes the load among them based on the platform used, the availability of free flows, and the predefined specific weight of each Selenoid.

Deploying GGR and GGR UI is easy:

• Install Docker.

• Create a directory for GGR configuration files mkdir -p /etc/grid-router/quota.

• Create a users.htpasswd file $ htpasswd -bc /etc/grid-router/users.htpasswd test test-password.

• Create a quotas file where you specify the address of the deployed Selenoid as the host:

$ cat /etc/grid-router/quota/test.xml
<qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru">
<browser name="android" defaultVersion="10.0" defaultPlatform="android">
  <version number="10.0">
    <region name="1">
      <host name="0.0.0.0" port="4444" count="1"/>
    </region>
  </version>
</browser>
</qa:browsers>

• Run the GGR container:

docker run -d \
  --name ggr \
  -v /etc/grid-router/:/etc/grid-router:ro \
  --net host aerokube/ggr:latest-release \
  -listen=:4445 -guests-allowed

• In the tests project, change the Appium port to the port of the running GGR:

val driver = AndroidDriver(URL("http://localhost:4445/wd/hub"), capabilities)

• Run the GGR UI container:

docker run -d \
  --name ggr_ui \
  -p 8888:8888 \
  -v /etc/grid-router/quota:/etc/grid-router/quota:ro \
  aerokube/ggr-ui:latest-release

• Run the Selenoid UI container where we pass the GGR UI port via selenoid-uri:

docker run -d \
  --name selenoid-ui \
  -p 4446:4446 \
  --link selenoid:selenoid \
  aerokube/selenoid-ui:1.10.4 \
  --selenoid-uri "<http://ggr-ui:8888>"

Our Selenoid UI should now display the status of all Selenoid clusters connected to GGR.

Now, we proceed to the running of tests on iOS

We use our own Mac mini farm to run UI tests on iOS. The farm can similarly be assembled from decommissioned but operational MacBooks. Alternatively, they can be rented. The following needs to be installed on each host:

• Appium (Appium 2.x must be used when one is working with iOS≥16).
• Xcode.

Some early problems

We were unable to replicate the structure used in running Android tests because we couldn’t find a way to run iOS simulators in Docker containers. One option that we considered was running Docker-OSX, but we ran into doubts about the legality of its use for any purposes unrelated to OS X Security Research. So, we decided to go a different route.

Iteration #1: GGR→Appium

We added Appium (port 4723) as a Selenoid host for iOS tests in the previously created GGR config file:

<qa:browsers xmlns:qa="urn:config.gridrouter.qatools.ru">
<browser name="android" defaultVersion="10.0" defaultPlatform="android">
  <version number="10.0">
    <region name="1">
      <host name="0.0.0.0" port="4444" count="1"/>
    </region>
  </version>
</browser>
<browser name="iPhone 14" defaultVersion="16.2" defaultPlatform="iOS">
  <version number="16.2">
    <region name="1">
      <host name="0.0.0.0" port="4723" count="1"/>
    </region>
  </version>
</browser>
</qa:browsers>

In such a case, the iOS scheme looks like this:

The structure used in this iteration is operational. The problem is that, in this case, we can only run tests in one workflow on each Mac mini, which is wasteful. Also, the cluster will not be displayed in the Selenoid UI.

Iteration #2: GGR→Selenoid→Appium

Selenoid allows you to work with more than just containers. The above problems informed our decision to use Selenoid when running iOS tests as well, though as an executable file:

• Download Selenoid (amd64/arm64).

• Create a browsers.json configuration file.

• Be sure to specify Appium and startup settings in the configuration file:

{
  "iPhone 14": {
    "default": "16.2",
    "versions": {
      "16.2": {
        "image": ["appium", "--log-timestamp", "--log-no-colors", ...]
      }
    }
  }
}

• Give permission to execute the Selenoid file. In our case, we used chmod 755.

• Run Selenoid via the terminal. We used the following parameters: selenoid -conf ~/browsers.json -disable-docker -capture-driver-logs -service-startup-timeout 4m -session-attempt-timeout 4m -timeout 6m -limit 2.

  • The specified timeouts are necessary because standard timeouts may not be sufficient for downloading the app from the cloud storage and launching the simulator.
  • The -limit parameter was used to set the maximum number of running simulators. This is the reference value to be used by the GGR in the future. The performance of the host is used as a guide for setting the parameter.
  • You can read more about startup settings in the Selenoid documentation.

• If need be, a PLIST file can be created on each Mac mini to autorun Selenoid in case of a sudden system restart.

Now, the process of the cluster looks like this:

With this approach, we partially achieve Selenoid UI functionality and the ability to run tests across multiple flows on the same host.

The downside is that, on each Mac mini, you have to manually carry out a multitude of routine tasks to create a simulator and link it with Appium by specifying the UUID and port assignment. This can become a problem if you need to upgrade to a new iOS version later.

Iteration #3: GGR→Selenoid→Bash→Appium

We have a large Mac mini farm that will continue growing as time goes on. With this in mind, we were looking for a way to make scaling easier so that we wouldn’t have to create simulators by hand and then connect them to Appium. With the previous schema in place, Appium and simulators would have had long lifetimes, which could have led to unpredictable consequences.

Searching for a solution, we discovered that a bash script can be specified as a host in the Selenoid configuration file:

{
  "iPhone 14": {
    "default": "16.2",
    "versions": {
      "16.2": {
        "image": ["~/bin/config/start_appium.sh", "iPhone 14"]
      }
    }
  }
}

This is what ours looks like:

#!/bin/bash

set -ex

DEVICE_NAME=$1
APPIUM_PORT=$(echo $2 | cut -d '=' -f 2)

function clean() {
  if [ -n "$APPIUM_PID" ]; then
      kill -TERM "$APPIUM_PID"
  fi
  if [ -n "$DEVICE_UDID" ]; then
      xcrun simctl delete $DEVICE_UDID
  fi
}

trap clean SIGINT SIGTERM

# Каждый симулятор имеет udid, поэтому чтобы запустить одинаковые устройства параллельно - клонируем и запускаем
# только клоны. Нельзя клонировать запущенное устройство. После закрытия сессии удаляем клон.
cloned_device_name="[APPIUM] ${DEVICE_NAME} ($(date +%Y%m%d%H%M%S))"
DEVICE_UDID=$(xcrun simctl clone "$DEVICE_NAME" "$cloned_device_name")

# https://github.com/appium/appium-xcuitest-driver#important-simulator-capabilities
WDA_LOCAL_PORT=$(($APPIUM_PORT+1000))
MJPEG_SERVER_PORT=$(($WDA_LOCAL_PORT+1000))
DEFAULT_CAPABILITIES='"appium:udid":"'$DEVICE_UDID'","appium:automationName":"'XCUITest'","appium:wdaLocalPort":"'$WDA_LOCAL_PORT'","appium:mjpegServerPort":"'$MJPEG_SERVER_PORT'"'

appium --base-path=/wd/hub --port=$APPIUM_PORT --log-timestamp --log-no-colors --allow-insecure=get_server_logs,adb_shell \
       --allow-cors --log-timestamp --log {choose_directory_for_logs} \
       --default-capabilities "{$DEFAULT_CAPABILITIES}" &
APPIUM_PID=$!

wait

If the script is used, pay close attention to the stated capabilities and Appium startup settings. These are set up here assuming that Appium 2.x is used for the run ‒ Appium 1.x does not require the vendor to be specified in capabilities, and there is no option of specifying --base-pat.

The script solves the problem of simulators running in parallel:

• When multiple Appiums are directly connected to GGR from one Mac mini, a problem arises with the emulators, because you cannot run the same emulator with identical UDIDs. You have to manually duplicate and hardcode the UDID each and every time. (For example, if you need to change the iOS version or the simulator model.)

• Poor scalability. It is necessary to run Appium manually every time, and regularly check it for conflicts with ports and simulators.

The use of Selenoid makes it possible to simplify this process down to a single script that does not create conflicts between multiple Appium + Simulator pairs within a single host. It launches Appium and kills it when receiving a corresponding signal from Selenoid, and it dynamically clones the simulators at startup and deletes them once the session ends.

The process we developed looks like this:

Next, we add the Selenoid addresses of each Mac mini to the configuration file of the deployed GGR, merging the Android and iOS structures:

Bottom-line summary

The assembled infrastructure allows us to run a total of about 500 UI tests per hour across 36 workflows on both platforms. Addition of a new host for Android tests can be fully automated by using the workflow on GitHub Actions, and takes about two minutes. In the very near future, we plan to automate the deployment of the Selenoid cluster on a Mac mini as well.

Further down the road, we would like to try out running Docker-OSX containers on a Mac mini with Linux to unify all the processes and make it easier to deploy to them without breaking any MacOS usage rules. If you have had any related experience, we'd love for you to share it in the comments.