I have a spark multimaster setup with zookeeper like this:
2 spark-master registered in zookeeper and two workers that register with the leader of the spark master, also the jobs are sent through livy which is a rest api.
My question is: how would load balancing be handled in ecs-fargate-elb considering that each component(spark-master,zookeeper,spark-workers,livy) is deployed in a separate ecs service.
Load balancing is not possible because spark master spark workers and livy connect via configuration files and the connection is created at compile time. The way in which the containers can communicate is through an interface that exposes the task definition called loopback interface that is fixed and has ip 127.0.0.1 and hostname localhost, thus the worker would register with the master thus spark:// 127.0.0.1:7077