question

If a service’s DynamoDb (DDB) is in a region from some users, is it better to call DDB from a lambda close to the end user OR close to the DDB?

(ie: if DDB is in us-east-1, but some users are in us-west-2, does it make sense to deploy a lambda in us-west-2 OR should they just call back to us-east-1)

note

This assumes that DDB Global Tables are not viable (ie: already have a global table in us-west-2, it probably would not make sense to replicate it into us-west-1).

Test

3 setups, for a ddb in us-east-1

  1. Lambda in us-west-2
  2. Lambda in us-east-1
  3. Cloudfront infront of us-east-1 Lambda (for edge routing)

Additionally a base line was generated by running the same test against a us-west-2 Lambda talking to a us-west-2 DDB.

The Lambda was a single js file running on nodejs-16.x with 125MB of memory and no provisioned capacity. The Lambda would perform a single GetItem request for the ‘350KB’ row in the table and return the result. This item was just an ID and a 350KB string attribute so as to generate some network transfer latency.

Process

From an ec2 in us-west-2, a 5 min load test with 30 concurrent users was run using artillery against each Lambda url / Cloudfront distribution.

Results

BASELINE
http.codes.200: ................................................................ 859
http.request_rate: ............................................................. 3/sec
http.requests: ................................................................. 859
http.response_time:
  min: ......................................................................... 108
  max: ......................................................................... 1836
  median: ...................................................................... 202.4
  p95: ......................................................................... 528.6
  p99: ......................................................................... 620.3

CROSS REGION DDB
http.codes.200: ................................................................ 914
http.request_rate: ............................................................. 3/sec
http.requests: ................................................................. 914
http.response_time:
  min: ......................................................................... 526
  max: ......................................................................... 3446
  median: ...................................................................... 1652.8
  p95: ......................................................................... 1978.7
  p99: ......................................................................... 2186.8
  
CROSS REGION LAMBDA
http.codes.200: ................................................................ 914
http.request_rate: ............................................................. 3/sec
http.requests: ................................................................. 914
http.response_time:
  min: ......................................................................... 167
  max: ......................................................................... 1936
  median: ...................................................................... 262.5
  p95: ......................................................................... 596
  p99: ......................................................................... 645.6
  
CLOUDFRONT INFRONT OF CROSS REGION LAMBDA
http.codes.200: ................................................................ 882
http.request_rate: ............................................................. 3/sec
http.requests: ................................................................. 882
http.response_time:
  min: ......................................................................... 202
  max: ......................................................................... 1859
  median: ...................................................................... 399.5
  p95: ......................................................................... 788.5
  p99: ......................................................................... 925.4

src

ran the same test from my home computer to see if starting outside of the AWS network would have any affect

it did not

Conclusion

Best to just call out to far away region directly (it had the closest latencies to the base line for all the stats)

notes

  1. repo with infra + lambda + test config is available at https://github.com/BASED-EDGE/cross-region-latency
  2. this only covers a limited use case, best to investigate for yourself

further questions

  • What about different services from just than just DDB (ie: RDS, S3, SNS)?
  • Does the same thing apply to DDB write requests? YES
  • What effect does using different regions have on the above results (ie: ca-central-1 to ap-southeast-2)
  • Why does remote DDB perform so much worse than remote lambda? the lambda is not transforming the data (apart from JSON.stringify()) so the network latency would be assumed to be very similar.
  • How would a cross region AWS PrivateLink endpoint change the results?