Unit Testing and System Testing? Are these the best methods we have?

 

In this post we will discuss some of the tests related concepts and guidelines. Some of the ideas here are contradicting with school based guidelines, but base on my experience these are usually the best method to ensure quality. I will specify in the post below whenever the guidelines contradicts with the common known testing methods.

Why do we need testing?

Delivering a product is a complicated process!

We need to go through the process of the following steps:

design, implementation, testing,  delivery

If we use Agile development methods, we'll go through these steps over and over again in each iteration.

Each of these steps could have mistakes and bugs. 

If we have no proper testing, the customer will suffer from our mistakes after the delivery.

We want to find the problems in the prior steps rather than in the later steps, since the cost of a problem in a prior step would affect all the following steps, and the fixing price would be much higher.

The testing step can include automatic testing and manual testing.

Both of these can be either manual or automatic.

Guess which one is better? ...

What is the difference between unit testing and system testing?

By school methodology the unit testing would test a single unit of code. It woill have zero dependencies with any entities external to the unit. These external entities include anything from network access, database access, files access, and up to other code unit access. The unit testing should mock any of these external items access with stubs, enabling the independent testing of the unit.

The system testing includes testing the system as a whole, even as a black box. These tests should simulate full scenario (of a user story) from it beginning to the final result. We will not use any stubs here, so we will need a full running system, including the database, persistence entities, network, and compute resources.

What are the pros and cons for unit testing vs. system testing?

Unit testing is created by the developer when implementing the specific unit of code. It requires development resources not only for implementing the code unit, but also to implement the unit test code, and to make the code unit adjustable behaviour by using stubs for any external APIs. This stub tailor work also complicates the code unit itself, as it should perprared to work with APIs (interfaces) for any external access. However, the benefits are a clear independent testing of a code unit. These tests are usually very fast due to the zero dependencies.

A major disadvantage in unit testing, is that we lake the ability to check a real integration of the code unit with any external entity, hence we can only write stubs that we think that simulate the real input/output to the code unit, and the expected flow of of code unit invocations.

System testing is mostly handled by dedicated automation team. It also requires development resources, but the costs of these resources is sometime lower. The system testing does not assume anything about the APIs between the code units, hence we get a really important advantage. However, the are problems with the system testing. It is developed by another team, so we need more knowledge transfer for any new feature. Each test must be run on a full system, hence it requires compute resources dedicated for each test, and we usually do not run tests in parallel on the system. We also suffer from the time limits. Running the tests on a real system is very very slow compared to unit testing. Not just that, think how can we check something that simulates analyzing data of last month? Do we run the tests for a full month?

What we can do? Project Testing!

Ok, this is where I diverge from the school methodology. It is up to you to think about it and decide if I am right. I can only testify from my years of experience that this had proven to be the best method I have ever used. It had some disadnatages, but the gain is high.

We want to have the benefits of the system testing AND the benefits of the unit testing. These are our goals:

• Testing should be fast
• Testing should not consume the full system compute resources
• Testing should not complicate the code
• Testing should check the real integration between code units
• Testing should not require additional team effort

How do we create project testing?

Project testing uses the same framework as the unit testing, but the difference is in its scope. We treat the entire project code as our single code unit. We still create mocks for external APIs: databases, files, network access, but any integration between the variuos code units is run as it run in the real system.

For that we will need to have all the code accessible to the project testing code. Either we use a mono repo, or we clone the git repositories to sub folders of the project testing code. 

The project testing code can check both internal processes (similar to the unit testing) that not a direct result of a user story scenario, and external processes (similar to the system testing) that simulate end-to-end full user story actions.

The goodies in this method are clear:

Do you want to move the time a month ahead? No problem, use a stub for the current time API.

Do you want to run tests in parallel? No problem, every test can run in its own process, and that all the compute resource it will use. No external entities such as databases are required.

Do we complicate the code? Much less than unit testing. Only external APIs are mocked, whose amount is usually much less than external APIs.

And we've done real code units integration tests, and no additional team effort was required.

Final Note

The project testing does not mean we need to stop using unit testing and system testing. Both of these are important, but we can invest more testing time resources in the project testing, and use less of the other methods.

Configuration VS. Code

 

In this post we will review a bad design pattern repeating itself in various products development projects. We will provide an example of such, and discuss how can we avoid it.

TL; DR

If the project configuration starts looking similar to code, use code not configuration.

Why is Configuration Required?

Many projects have configuration files. The configuration in the file can dynamically change the code behaviour. 

Why do we keep configuration? An alternative would be to use consts, and ask our code user to modify the consts, and to write another code that will use our code. But this is something we cannot ask from an end user which is not a software engineer, so we use configuration.

Right?

Always right?

Sample Project

Let have an example of a real project that I was part of. The project is a general installation management for various components in the company. Each component requires some registry settings that are read from the user, and some installation scripts. Each component might be depdend of other compoents, In addition, we need to manage a list of what is installed, and where.

Hey! I have an idea! Let make it an XML configuration file:

<component>
  <name>OS Agent</name>
  <dependency>Memory Agent</dependency>
  <input>
    <text>serverName</text>
    <text>installationFolder</text>
  </input>
  <installationScript>install_os_agent.sh</installationScript>
</component>

Look great, right?

Well, yes. Now anyone can use our installer by simply creating a configuration file. No code is required. That's great. Lets send it to our company users!

Oh... There is a requirement from the Redis Agent that in case the redis type is cluster, we should run another script. 

Mmm... No problem, we can add simple logic in the XML. Here we go:

<component>
  <name>Redis Agent</name>
  <dependency>OS Agent</dependency>
  <input>
    <text>serverName</text>
    <text>installationFolder</text>
    <text>redisType</text>
  </input>
  <installationScript>install_redis_agent.sh</installationScript>
  <if type="equal">
    <input>redisType</input>
    <const>cluster<const>
    <do>
      <installationScript>install_redis_cluster.sh</installationScript>
    </do>
  </if>
</component>

Great! We did it.

Oh.. The kubernetes agent requires us to loop on all the servers list, and run a script for each on.

Sure, we can do that:

<component>
  <name>Kubernetes Agent</name>
  <input>
    <text>servers</text>
  </input>
  <installationScript>install_kubernetes_agent.sh</installationScript>
  <loop>
    <input>servers</input>
    <do>
      <installationScript>add_server.sh</installationScript>
    </do>
  </if>
</component>

Wait, some of the components are using the exactly the same list of installation scripts, let's not repeat the same configuration. Let use includes:

<component>
  <name>Grafana Agent</name>
  <input>
    <text>servers</text>
  </input>
  <installationScript>install_grafana_agent.sh</installationScript>
  <include>common/common_scripts.xml</include>
</component>

One last thing: some of the teams said that the configuration is great, but it is a bit complicated. Would it be possible to create a GUI to manage the configuration? A wizard to create actions in the configuration, and to edit existing configurations will be great...

End of Story

So, you end up rewriting a new programming language in XML. Anyone who want to use this requires eduaction resources (time and teachers). Debugging issues in the configuration is possible only by the team who owns this code. Any small improvement which is built-in within programming languages requires development time. This is a living hell. How did we got here?

How to Avoid?

To avoid this. Once we identify logic in our configuration that control the actual configuration we should stop, and rethink the design. In most cases the good solution would be to supply a code library that other teams could use as libraries in their own code. No more code in a configuration file, but codein a programming language that anyone knows, and can debug.

While it seems quite obvious when reading this post, this mistake repeats itself over and over again. It is very tempting to avoid code and have a full configurable component that does anything you need, but this is a wrong point a view. We should treat projects as tools, expert at their domain, but nothing else. Other functionalities should be owned by the tool users, hence enabling them with any possible customization they need.

Attestation Validation on Server Side using Java

Attestation Posts:

• Android Device Attestation Client Side and Server Side using SafetyNet
• Android Device Attestation Client Side and Server Side using Play Integrity
• Attestation Validation on Server Side using Java

In this post we will review server side validation of an attestation token sent from an android device.

The validation will be done by a java based service.

First, we add a dependency in the pom.xml for Google's play integrity API:

<dependency>
   <groupId>com.google.apis</groupId>
   <artifactId>google-api-services-playintegrity</artifactId>
   <version>v1-rev20220211-2.0.0</version>
</dependency>

Now we can validate the attestation token:

package com.shieldsquare.ruleslib.rules.features;

import com.google.api.client.googleapis.services.GoogleClientRequestInitializer;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.playintegrity.v1.PlayIntegrity;
import com.google.api.services.playintegrity.v1.PlayIntegrityRequestInitializer;
import com.google.api.services.playintegrity.v1.model.DecodeIntegrityTokenRequest;
import com.google.api.services.playintegrity.v1.model.DecodeIntegrityTokenResponse;
import com.google.auth.http.HttpCredentialsAdapter;
import com.google.auth.oauth2.GoogleCredentials;

import java.io.ByteArrayInputStream;
import java.util.List;

public class AttestationServer {

  public static void validate(String token,
                              String nonce,
                              String sidCredentials) throws Exception {
    DecodeIntegrityTokenRequest requestObj = new DecodeIntegrityTokenRequest();
    requestObj.setIntegrityToken(token);
    GoogleCredentials credentials = GoogleCredentials.fromStream(new ByteArrayInputStream(sidCredentials.getBytes()));
    HttpRequestInitializer requestInitializer = new HttpCredentialsAdapter(credentials);

    HttpTransport HTTP_TRANSPORT = new NetHttpTransport();
    JsonFactory JSON_FACTORY = new JacksonFactory();
    GoogleClientRequestInitializer initializer = new PlayIntegrityRequestInitializer();

    PlayIntegrity.Builder builder = new PlayIntegrity.Builder(HTTP_TRANSPORT, JSON_FACTORY, requestInitializer)
        .setApplicationName("my-application")
        .setGoogleClientRequestInitializer(initializer);
    PlayIntegrity play = builder.build();

    DecodeIntegrityTokenResponse response = play.v1().decodeIntegrityToken("com.example.myapp", requestObj)
        .execute();

    boolean valid = false;
    if (response.getTokenPayloadExternal().getRequestDetails().getNonce().equals(nonce)) {
      List<String> verdict = response.getTokenPayloadExternal().getDeviceIntegrity().getDeviceRecognitionVerdict();
      valid = verdict.contains("MEETS_DEVICE_INTEGRITY");
    }

    // do something with the result
    System.out.println("token valid:" + valid);
  }
}

The token is received at the end device, and sent to the server side for validation

The nonce is expected to be generated by the server and sent to the client, as a kind of challenge. Alternatively, the nonce can be some kind of unique identifier of the application on the device, which is not expected to be used in other devices.

The sid credentials is the content of a json file, with credentials for service account that has permissions to validate the attestation token.

Notice that the is a limit for the free token validation calls, and after that you'll need to pay for each call, hence make sure to balance between validation requirements and costs.

Build iOS Framework using Xcode CLI

 

In this post we will use a script to create an Xcode library using the xcode CLI.

We start by installing the Xcode CLI tool:

xcode-select --install

In some cases you will need to point to the location of the Xcode, so instead, you will need to run the following command:

sudo xcode-select -s /my_apps/Xcode.app/Contents/Developer

Next, change directory to the root of the library and run the following script:

FRAMEWORK_NAME=MyFramework
SIMULATOR_ARCHIVE_PATH=./Output/${FRAMEWORK_NAME}-iphonesimulator.xcarchive
IOS_DEVICE_ARCHIVE_PATH=./Output/${FRAMEWORK_NAME}-iphoneos.xcarchive
FRAMEWORK_PATH=./Output/${FRAMEWORK_NAME}.xcframework

xcodebuild archive -scheme ${FRAMEWORK_NAME} -destination="iOS Simulator" -archivePath "${SIMULATOR_ARCHIVE_PATH}" -sdk iphonesimulator SKIP_INSTALL=NO BUILD_LIBRARIES_FOR_DISTRIBUTION=YES
xcodebuild archive -scheme ${FRAMEWORK_NAME} -destination="iOS" -archivePath "${IOS_DEVICE_ARCHIVE_PATH}" -sdk iphoneos SKIP_INSTALL=NO BUILD_LIBRARIES_FOR_DISTRIBUTION=YES
xcodebuild -create-xcframework \
    -framework ${SIMULATOR_ARCHIVE_PATH}/Products/Library/Frameworks/${FRAMEWORK_NAME}.framework \
    -framework ${IOS_DEVICE_ARCHIVE_PATH}/Products/Library/Frameworks/${FRAMEWORK_NAME}.framework \
    -output "${FRAMEWORK_PATH}"

This runs for ~2 mintues, and finally create an xcframework file which can be used by other applications.

Using Gyroscope and Accelerometer Sensors

 

In this post we will review how to use the gyroscope and the accelerometer sensors in an android application. These sensors are part of the motion sensors available on some devices.

Sensors on an android application are very easy to use. The following class registers the sensors, and updates its fields with the related sensor coordinates.

package com.my.application;

import android.content.Context;
import android.hardware.Sensor;
import android.hardware.SensorEvent;
import android.hardware.SensorEventListener;
import android.hardware.SensorManager;

public class Sensors {
    float accelerometerValueX = -9;
    float accelerometerValueY = -9;
    float accelerometerValueZ = -9;

    float gyroscopeValueX = -9;
    float gyroscopeValueY = -9;
    float gyroscopeValueZ = -9;


    public void registerReceivers() {
        // get context from the app
        Context context = ...;
        SensorManager manager = (SensorManager) context.getSystemService(Context.SENSOR_SERVICE);
        final Sensor gyroscopeSensor = manager.getDefaultSensor(Sensor.TYPE_GYROSCOPE);
        final Sensor accelerometerSensor = manager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER);
        SensorEventListener listener = new SensorEventListener() {

            @Override
            public void onAccuracyChanged(Sensor arg0, int arg1) {
                // Do nothing
            }

            @Override
            public void onSensorChanged(SensorEvent event) {
                Sensor sensor = event.sensor;
                try {
                    if (sensor.getType() == Sensor.TYPE_GYROSCOPE && (gyroscopeSensor != null)) {
                        gyroscopeValueX = event.values[0];
                        gyroscopeValueY = event.values[1];
                        gyroscopeValueZ = event.values[2];
                    } else if (sensor.getType() == Sensor.TYPE_ACCELEROMETER && (accelerometerSensor != null)) {
                        accelerometerValueX = event.values[0];
                        accelerometerValueY = event.values[1];
                        accelerometerValueZ = event.values[2];
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }

        };
        if (gyroscopeSensor != null) {
            manager.registerListener(listener, gyroscopeSensor, SensorManager.SENSOR_DELAY_NORMAL);
        }
        if (accelerometerSensor != null) {
            manager.registerListener(listener, accelerometerSensor, SensorManager.SENSOR_DELAY_NORMAL);
        }
    }

}

While getting the sensors coordinates is easy, analyzing their meaning should be done carfully.

The accelerometer coordinates always "feel" the gravity, hence the actual values also include the gravity force. In case of need, the gravity force can be substructed from the accelerometer coordinates.

Using the sensors we can change the behavior of an application to reflect real world activities of the user. We can even track these sensors to try to figure what is the user doing, whether the user is at rest, walking, playing tennis, running, or even swimming. This infomation might even unknowingly break the user privacy, so we need to be careful about storing this information.