Address data processing outside the data sovereignty region (GDPR)

The high level of cloud adoption brings closer and closer to us the day to day problems that were easily solved using on-premises solutions. The data regulations related to data sovereignty and specific region and country regulations like GDPR add an extra complexity layer to the application we are building inside the cloud.

We have an imaginary organization called Osotnoc that is active in Australia, the US, the UK and the EU. Each country has strict data regulations that require Osotnoc to store customers data inside each region.

Microsoft Azure has a strong presence in each region, so the team is building a solution on Azure SQL and App Service. The solution is deployed in all 4 regions, fully compliant with all local requirements but pretty expensive. Except for the database and computation layer, 4 different instances of Azure Application Gateways are deployed with WAF on top of them.

Besides this, more than 30% of active users require access to data from other regions. The current solution requires them to specify the tenant/region they want to access during the login step. To have the ability to compare data across regions, they need to log into two separate browsers. It’s not a nice UX.

But what if we could make the solution more simple? The local data requirements are mainly referring to the data storage layer and not to the computation. Meaning that nobody stops us from getting content from repositories that are from different regions.

One way to do this is to use the sharding capability of Azure SQL Database and the Shard Map Manager. This would enable the application layer to have the ability to manage from which shard (DB) to connect. It allows users to run the same query across multiple regions (shards) and display them on the UI.

The solution works pretty well, but some logic needs to be handled at the application layer. It is not complicated, but an extra effort is required to support this. The good thing is that we have support for .NET and Java, the ShartMapManager, part of the Elastic Database package that can be used with success for this purpose.

There is an impact on latency when you need to load content from databases that are in a different region, but you can provide a better experience for users in this way. It’s a good tradeoff that you can do.

Depending on the regulations and the user patterns, you could even add a cache layer in each region for data that is often accessed. A cache is not seen as long term storage, and you are allowed to use it in specific situations, depending on the content type and the level of encryption that you use. The downside with a cache solution is that they are one of the weakest parts of a system. The security layer exists but is not their main straight. In the end in most cases, a CDN or a remote cache is used for non-sensitive content.

In both cases (CDN or cache) you need to have content full encrypted, the encryption key stored in the EU and clear procedures related to how you manage data and what kind of actions you take.

What you need to take into account from data compliance and GDPR point of view:

• Processing EU citizen data outside the EU region is allowed as long as the encryption key is stored in the EU. Special conditions are required to be allowed to do something like this
• An encrypted cache outside the EU can be used, but content needs to be encrypted and additional technical and organizational measures need to be taken.