Test and Coverage for Go Modules

 

In this post we will review how to run tests, and check coverage for Go modules.

When creating a Go module, we add unit tests for anything we can, to avoid a bug from being detected only at system test stage, or worse at production.

To find which part of the code is tested, we can use the Go builtin tools:

go test ./... -race -covermode=atomic -coverprofile=../coverage.out
go tool cover -html=../coverage.out

The first command runs the tests, including check of a race condition issue, and prints the following output:

?   	example.com/app/detector/server/detectionstatuses	[no test files]
ok  	example.com/app/detector/server/detector	2.092s	coverage: 98.6% of statements
ok  	example.com/app/detector/server/extractor	0.240s	coverage: 91.3% of statements
?   	example.com/app/detector/server/featuresnames	[no test files]
ok  	example.com/app/detector/server/headersparser	0.027s	coverage: 100.0% of statements

For each package we get a indication whether the tests are OK/FAIL, and the coverage percent of the tests.

The second command, opens a browser with specific lines display. This enables us to find the code lines that are not tested.

Notice the coverage is considering only the current package tests, so if we have package a tests using package b code, b code will not be considered as a covered code.

Android Device Attestation Client Side and Server Side using Play Integrity

Attestation Posts:

• Android Device Attestation Client Side and Server Side using SafetyNet
• Android Device Attestation Client Side and Server Side using Play Integrity
• Attestation Validation on Server Side using Java

 

In the previous post, we've presented mobile device attestation using Google's SafetyNet. However, in recently Google announced disontinue of SafetyNet. Hence we've updated our client side and server side to use PlayIntegrity instead.

Unlike SafetyNet, PlayIntegrity documentation site is extremely poor, and the migrate was frustrating. Some of the insights about how PlayIntegrity were revealed in this stackoverflow post (but it does the attestation on the mobile device only). Other insights were only a matter of trying various APIs with zero documentation.

I sum the steps done below, and hope that in the future, the documentation will get better.

Client Side

First, add dependencies:

dependencies {
    implementation 'com.google.android.play:integrity:1.0.1'                  
    implementation 'com.google.apis:google-api-services-playintegrity:v1-rev20220211-1.32.1'
    implementation 'com.google.api-client:google-api-client-jackson2:1.20.0'
    implementation 'com.google.auth:google-auth-library-credentials:1.7.0'
    implementation 'com.google.auth:google-auth-library-oauth2-http:1.7.0'
}

Next implement token request on the client side, and send it to your backend server.

Project ID can be located as explained here.

import android.content.Context;

import com.google.android.gms.tasks.Task;
import com.google.android.play.core.integrity.IntegrityManager;
import com.google.android.play.core.integrity.IntegrityManagerFactory;
import com.google.android.play.core.integrity.IntegrityTokenRequest;
import com.google.android.play.core.integrity.IntegrityTokenResponse;

public class Attestation {

    public void start(String nonce) {
        IntegrityManager integrityManager = IntegrityManagerFactory.create(context);

        Task<IntegrityTokenResponse> integrityTokenResponse = integrityManager
                .requestIntegrityToken(IntegrityTokenRequest
                        .builder()
                        .setNonce(nonce)
                        .setCloudProjectNumber(YOUR_GCP_PROJECT_NUMBER_HERE)
                        .build());

        integrityTokenResponse.addOnSuccessListener(response -> {
                String token = response.token();
                ExecutorService executor = Executors.newSingleThreadExecutor();
                executor.execute(() -> {
                    // send the token to your backend server
                });
        });
    }


}

Server Side

On the server side, we get the token sent from the client, and ask Google's verdict for the token.

In this case, we assume the device is valid in case the verdict includes MEETS_DEVICE_INTEGRITY.

To enable access to Google services, we create an account credentials json file(see instructions in the stackoverflow post), and send it in the GoogleCredentialsPath variable.

package attestation

import (
 "context"
 "fmt"
 "google.golang.org/api/option"
 "google.golang.org/api/playintegrity/v1"
)

func  Execute()  error {
 request := AttestationRequest{}
 err := GetBody(&request)
 if err != nil {
  return  fmt.Errorf("get body failed: %v", err)
 }

 ctx := context.Background()
 playIntegrityService, err := playintegrity.NewService(ctx, option.WithCredentialsFile(core.Config.GoogleCredentialsPath))
 if err != nil {
  return  fmt.Errorf("create play integrity service failed: %v", err)
 }

 tokenRequest := playintegrity.DecodeIntegrityTokenRequest{
  IntegrityToken:  request.Token,
  ForceSendFields: nil,
  NullFields:      nil,
 }

 tokenCall := playIntegrityService.V1.DecodeIntegrityToken("your.client.java.package.prefix.com", &tokenRequest)
 response, err := tokenCall.Do()
 if err != nil {
  return  fmt.Errorf("token call failed: %v", err)
 }

 if response == nil {
  return fmt.Errorf("response is null")
 }

 payload := response.TokenPayloadExternal
 if payload == nil {
  return fmt.Errorf("payload is null")
 }

 integrity := payload.DeviceIntegrity
 if integrity == nil {
  return  fmt.Errorf("integrity is null")
 }

 for _, verdict := range integrity.DeviceRecognitionVerdict {
  if verdict == "MEETS_DEVICE_INTEGRITY" {
   return nil
  }
 }

 return fmt.Errorf("verdict is %v", integrity.DeviceRecognitionVerdict)
}

Final Note

In this post we have used Google PlayIntegrity to validate a mobile device.

We presented the steps for client side and server side implementation.

Make sure to check quotas for the allowed attestation requests before moving to production environment.

Android Device Attestation Client Side and Server Side using SafetyNet

Attestation Posts:

• Android Device Attestation Client Side and Server Side using SafetyNet
• Android Device Attestation Client Side and Server Side using Play Integrity
• Attestation Validation on Server Side using Java

In this post we will review the steps to perform an android device attestation.

This includes the client side code to call google's safetynet and receive a token, and the server side to validate the token.

A good guide for this is included in the SafetryNet Attestation API documentation. The following class runs the entire flow for attestation client side.

package com.example.app;

import android.content.Context;
import android.os.AsyncTask;
import androidx.annotation.NonNull;

import com.google.android.gms.common.ConnectionResult;
import com.google.android.gms.common.GoogleApiAvailability;
import com.google.android.gms.safetynet.SafetyNet;
import com.google.android.gms.safetynet.SafetyNetApi;
import com.google.android.gms.tasks.OnFailureListener;
import com.google.android.gms.tasks.OnSuccessListener;

public class DeviceAttestation extends AsyncTask
    implements OnSuccessListener<SafetyNetApi.AttestationResponse>,
     OnFailureListener
    {

    private String API_KEY = "AIzaSyCwl123456789012345678934567dsqIqY";

    private byte[] nonce;

    public Attestation(byte[] nonce) {
        this.nonce = nonce;
    }

    public void start() {
        Context context = Bouncer.getInstance().getContext();
        if (GoogleApiAvailability.getInstance().isGooglePlayServicesAvailable(context)
                != ConnectionResult.SUCCESS) {
            throw new RuntimeException("too old Google Play Services version");
        }

        SafetyNet.getClient(context).attest(this.nonce, API_KEY)
                .addOnSuccessListener(this)
                .addOnFailureListener(this);
    }

    @Override
    public void onSuccess(SafetyNetApi.AttestationResponse response) {
        String token = response.getJwsResult();
        this.execute();
    }

    @Override
    protected Object doInBackground(Object[] objects) {
        sendTokenToServer(this.token);
        return null;
    }

    @Override
    public void onFailure(@NonNull Exception e) {
        Logger.warn("attestation failed", e);
    }

}

The attestation flow is a follows:

• Call the Attestation constructor with a nonce. The nonce should be a pretty random value, even best to make it unique for the device, for example: the user name.
• Call to start() method, which starts the communication with Google's attestation server.
• Once completed the onSuccess() method is called, and the attestation token is received.
• The onSuccess uses ASyncTask to start communication with the application backend, where the attestation token is delivered.
• The server then validates the attestation token, and decides if to allow the client to proceed or not.

On the server side, we use Go to validate the attestation token.

package attestation

import (
 "fmt"
 "github.com/wongnai/safetynet/v2"
)

type Executor struct {
}


func (e *Executor) Execute(p web.Parser) (interface{}, error) {
 request := AttestationRequest{}
 err := p.GetBody(&request)
 if err != nil {
  return nil, fmt.Errorf("get body failed: %v", err)
 }


 attestation, err := safetynet.Validate(request.Token)
 if err != nil {
  return nil, fmt.Errorf("validate attestation failed: %v", err)
 }

 if !attestation.CTSProfileMatch {
  return nil, fmt.Errorf("attestatation failed: %+v", attestation)
 }


 return AttestationValidResponse{}, nil
}

We use the CTSProfileMatch to decide if the device is a valid device, or a bot emulator, and allow or fail to client side to continue.

Final Notes

We have presented client side and server side for attestation on android devices.

When moving to production, we must consider the attestation quota, as well as whether we want to use not only CTSProfileMatch, but also basicIntegrity to allow access.

AWS EFS, and why it is a bad solution for AWS Batch

 

In this post we will review an disk space issue that I've encountered, and the findings I've discovered in the process.

TL;DR
AWS EFS seems like a great solution for disk space issues, but it is actually relevant only for long lived persistence used mostly with read operations. 

In the previous post, I've reviewed usage of AWS batch, and automation of it using python's boto3 library. As the project continued, we've used the AWS batch to analyze portions of big data stored in AWS S3. While testing the analyze process as a docker image running on my local development machine, everything worked great. But when running it on the AWS Batch based on Fargate, we encountered out disk space issues. So, I went and looked for AWS Batch configuration to increase the disk space requirement for the container. It seems that AWS Fargate does support this up to 200G, as specified here.

However, AWS development team (so far) did not add this option as part of the AWS Batch job definition. So I went to look for an alternative, and found the AWS EFS, which seems like a great solution: unlimited storage, automatically allocated per your need, pay only per what you use.

So, I've created a new EFS, and updated the AWS batch job definition to use the EFS, and restarted the AWS jobs. 

... but ... wait... why are jobs stuck for so long??

Digging into the AWS EFS documentation, I've located the EFS quotas guide. The quotas limit the amount of read and write operations on the AWS EFS according to the size of the used space on the EFS. Once you hit this quota, you're stuck until the next timeframe you get an additional quota, use it, and get stuck again. 

In my case I've had used the EFS as a temporary storage for cache data, and write and read in each AWS batch job about 200GB multiple times. This means that the jobs are stuck very quickly.

The solution would be to provision in advanced the required quota for read and write operations. But the price for this is exteremly high, as if you would have used the disk space that would had been used to get this quota.

Eventually, I've gave up on the disk space caching mechanism, and changed the algorithm to use much less storage, on the downside of slower processing.

Final Note

First, I hope that AWS team would add the support for speficying disk space requirements for AWS batches. This would surely simlify the whole mess described here.

Second, AWS EFS seems to be unfit for short live storage used for many read and write operations. It is more long lived persistence that is mostly read, use low rate of read/write operations.