Support for other SMC protocols in MP-SPDZ

[Quantum-SMC]

Hi,
Can other types of protocols in MP-SPDZ (e.g. MASCOT) be integrated to your platform? I need to use a protocol that utilises OT.
Thank you.

[Tobias512]

Hi Quantum-SMC,

first off, I don't know how many protocols provide by MP-SPDZ can be used to run our code. We only have tested it for the four protocols specified in the README. Most protocols should work because of the way MP-SPDZ is designed.

I will give you some directions on how to integrate other protocols so you can try out the protocols you need.
First, you need to extend get_protocol which return the shell script name of the protocol based on the command line argument --protocol.

https://github.com/encryptogroup/SAFEFL/blob/31773ec23e25e620b6662ae284b507f1fffcd6e7/main.py#L132C1-L158C34

Second, you may need to change the compilation arguments used to compile the MP-SPDZ code. Currently, we are using -R 64 -X, but this may not work with other protocols.

SAFEFL/main.py

Line 380 in 31773ec

os.system('./compile.py -R 64 -X ' + args.filename_server + ' ' + str(args.port) + ' ' + str(num_params) + ' ' + str(num_gradients) + ' ' + str(args.niter) + ' ' + str(args.chunk_size) + ' ' + str(args.threads) + ' ' + str(args.parallels))

Other than that, I don't think that more changes are needed to integrate new protocols into the framework.

[Quantum-SMC]

Thank you for your reply. It seems working. However, while python main.py I get the following errors. It seems that some files are missing and I couldnt find them nether in MP-SPDZ nor in SAFEFL . I appreciate your help:

File "/Users/zeinab/Downloads/SAFEFL/mpspdz/Compiler/program.py", line 256, in init_names raise CompilerError( Compiler.exceptions.CompilerError: found none of the potential input files: '14000', 'Programs/Source/14000.mpc', 'Programs/Source/14000.py'
...

Running /Users/zeinab/Downloads/SAFEFL/mpspdz/Scripts/../mascot-party.x 0 mpc_fedavg_server-14000-3372-30-2000-200-1-1 -pn 13248 -h localhost -N 2
Running /Users/zeinab/Downloads/SAFEFL/mpspdz/Scripts/../mascot-party.x 1 mpc_fedavg_server-14000-3372-30-2000-200-1-1 -pn 13248 -h localhost -N 2
libc++abi: terminating due to uncaught exception of type file_error: File Error : Missing 'Programs/Schedules/mpc_fedavg_server-14000-3372-30-2000-200-1-1.sch'. Did you compile 'mpc_fedavg_server-14000-3372-30-2000-200-1-1'?

[Tobias512]

I just tried it out and got Mascot working. I believe you made an error in the compilation I highlighted above:

SAFEFL/main.py

Line 380 in 31773ec

os.system('./compile.py -R 64 -X ' + args.filename_server + ' ' + str(args.port) + ' ' + str(num_params) + ' ' + str(num_gradients) + ' ' + str(args.niter) + ' ' + str(args.chunk_size) + ' ' + str(args.threads) + ' ' + str(args.parallels))

According to the error, MP-SPDZ is looking for a file named 14000.mpc or 14000.py to compile it. I think you accidentally deleted args.filename_server from the line above, which is the first variable argument given to compile.py. This argument specifies the file to compile and should be either mpc_fedavg_server or mpc_fltrust_server. In your case, the first argument is 14000 which is the default port, which leads me to believe the correct filename is missing.

In my case, I got MASCOT working by adding the protocol like you did and changing the -R to a -F in the compilation string shown above.

I hope this fixes your problem.

[Quantum-SMC]

Thank you. I was able to run main.py for Mascot. However, it keeps waiting for the other clients to connect (according to MP-SPDZ, I might need to open a new terminal? ). Could you please tell me how to fix this:

/opt/homebrew/Caskroom/miniconda/base/envs/safefl_env/bin/python /Users/zeinab/Downloads/SAFEFL/main_working_for_mascot.py 
cpu
dataset: p0.1_HAR, server_pc: 100, bias: 0.5, nworkers: 30, net: lr, niter: 2000, lr: 0.25, batch_size: 64, nbyz: 6, attack: no, aggregation: fedavg, FLOD_threshold: 0.5, Flame_epsilon: 3000, Flame_delta: 0.001, Number_runs: 1, DnC_niters: 5, DnC_c: 1, DnC_b: 2000, MP-SPDZ: True, Port: 14000, Chunk_size: 200, Protocol: mascot, Threads: 1, Parallels: 1, Seed: 1, Test Every: 50
LinearRegression(
  (linear): Linear(in_features=561, out_features=6, bias=True)
)
Setting up SSL for 2 parties
..+...+.+.........+...+..+......+.+......+..+++++++++++++++++++++++++++++++++++++++*..+.........+..+++++++++++++++++++++++++++++++++++++++*...+..+............+......+....+...+...+.........+......+.....................+...+......+......+...............+......+..+...+...+....+..+.+..+...+...+.......+.....+...+.......+..+..........+..+.........+....+...+............+.....+.+.....+......+...+......+.+......+..+............+.+...+..+.......+.....+.+..+....+...........+............+...+....+...+......+.....+.+.....+......+...+....+..+.........+.+..................+...+............+.....+...............+...+....+.....+.+..+......+................+.....+......+....+...+..+.........+.......+...+..+....+.....+...+.+..+.......+......+..+....+........................+......+...+...+..............+.+.....+..........+.................+.+.....+.+........+......+.......+.................+.+......+.....++++++
.....+....+++++++++++++++++++++++++++++++++++++++*....+...+....+...+..............+...+.......+.....+.............+++++++++++++++++++++++++++++++++++++++*.+...+......+............+.........+.+.....+.......+.....+.+..+.........+.........+....+.....+......+....+.....................+.....+.+...+.....+......+......+..................+...............+.+.....+.......+..+......+......+.+..+.+......+......+...+.....+.+...........++++++
-----
...+......+........+.+..+....+.....+...+.......+.........+..+.......+..+.+++++++++++++++++++++++++++++++++++++++*.....+..+++++++++++++++++++++++++++++++++++++++*.+...+......+..+...+.+......+........+....+...+..+.+..............+.+...........+.+........+.......+.....+...+......+.....................+....+........+..........+.....+.......+.........++++++
.......+.+......+.....+.+.....+.........+.+..............+...+...+....+...+......+............+.....+...+.+...............+.....+++++++++++++++++++++++++++++++++++++++*...+.....+......+.......+..+.........+++++++++++++++++++++++++++++++++++++++*..+...+.+.....+.+......+.....+.......+...........+.......+..+...+..................+......+..................+.......+...+..++++++
-----
Doing Player-Data
.........+++++++++++++++++++++++++++++++++++++++*...+..+.+......+......+.....+...+.+...+...........+.+...+...+++++++++++++++++++++++++++++++++++++++*........+.......+..+...+.............+...+..+.............+......+........+..........+........+.......+..+.+..............+.+..+......+...............+.......+........++++++
...+...............+...+...+..+++++++++++++++++++++++++++++++++++++++*..+++++++++++++++++++++++++++++++++++++++*.....+......+...+.+..................+......+..+.........+...+.+.........+......+..+............+.+............+..+............+....+......+........+..........+........+...+.......+.....+.......+.....+.+.....+.+...+......+.....++++++
Setting up SSL for 1 parties
-----
Doing Player-Data
Data done
Starting Computation Parties
Running /Users/zeinab/Downloads/SAFEFL/mpspdz/Scripts/../mascot-party.x 0 mpc_fedavg_server-14000-3372-30-2000-200-1-1 -pn 18586 -h localhost -N 2
Running /Users/zeinab/Downloads/SAFEFL/mpspdz/Scripts/../mascot-party.x 1 mpc_fedavg_server-14000-3372-30-2000-200-1-1 -pn 18586 -h localhost -N 2
Using statistical security parameter 40
Arguments:
Programm: mpc_fedavg_server
Iterations: 2000
Workers: 30
Parameters: 3372
Portnum: 14000
Chunk Size: 200
Threads: 1
Parallel: 1
Listening for client connections on base port 14000
Starting aggregation
Data send

Regards,

[Tobias512]

Unfortunately, I don't know what exactly is the cause of your problems. You should not need to do anything besides run main.py and the console output looks as expected for running main.py.

If I run main.py --mpspdz --protocol=mascot I get the following output:

/home/tobias/anaconda3/envs/safefl/bin/python /home/tobias/Git/SAFEFL/main.py --mpspdz --protocol=mascot 
cpu
dataset: p0.1_HAR, server_pc: 100, bias: 0.5, nworkers: 30, net: lr, niter: 2000, lr: 0.25, batch_size: 64, nbyz: 6, attack: no, aggregation: fedavg, FLOD_threshold: 0.5, Flame_epsilon: 3000, Flame_delta: 0.001, Number_runs: 1, DnC_niters: 5, DnC_c: 1, DnC_b: 2000, MP-SPDZ: True, Port: 14000, Chunk_size: 200, Protocol: mascot, Threads: 1, Parallels: 1, Seed: 1, Test Every: 50
LinearRegression(
  (linear): Linear(in_features=561, out_features=6, bias=True)
)
Setting up SSL for 2 parties
..+++++++++++++++++++++++++++++++++++++++*..+.+.....+++++++++++++++++++++++++++++++++++++++*.........+.+............+...+...........+...+.......+...+..+...............+................+...+..............+.......+......+.....+..........+...............+..+.+.........+............+......+............+..+...+.......+...+..+...+...+...+.......+..+.+..+...+....+.....+.......+......+...+.....+....+..+...+.......+.....+..................+...+.........+...+......+....+.....+...+....+........+.............+.....+...++++++
....+................+.....+....+..+.........+.+...........+.......+....................+.+..+...+.......+..+.+....................+.+........+..........+...........+++++++++++++++++++++++++++++++++++++++*....+....+......+..+......+.+......+..+.+..+....+++++++++++++++++++++++++++++++++++++++*..........+..................++++++
-----
..+....+++++++++++++++++++++++++++++++++++++++*...+...+.......+........+......+.......+..+......+++++++++++++++++++++++++++++++++++++++*.........+....+......+.........+..+...+..........+........+....+...+.....+.+.....+.......+..+..........+..+.+..+...+..........+...+...........+.+..............+......+...+.+...........+...+.+.....+.+.........+......+..+.+.........+..+.+............+...+....................+.......+..+...+.............+.........+...............+.....+....+......+...+.....+.......+......+...+.....+......+..........+.........+.....+.........+...+....+.....+.........+......+...+......+.........++++++
.....+..+++++++++++++++++++++++++++++++++++++++*.+++++++++++++++++++++++++++++++++++++++*...........................+..+.+..+.+......+...+...............+...+...+..+..........+...+...............+..+....+.....+...+............+.+.............................+....+......+...+............+...+..+...............+.+..+.+...........+.+..+..........+...+........+......+...+..................+.......+...+..+.+.....+......+...+.+...........+.....................+....+.....+.+............+........+............+...+...+....+........+....+........+.+.....+.......+...+..+.........+.............+...+.....+...+.........................+.....+............+.......+..+...+......................+......+............+...+........+....+.....+.++++++
-----
Doing Player-Data
Setting up SSL for 1 parties
..+......+.+........+.........+.+.................+...+.+...............+......+++++++++++++++++++++++++++++++++++++++*..+.....+...+.+.....+......+...+++++++++++++++++++++++++++++++++++++++*......+..+..........+........+.......+.........+.....+.........+......+.+..+.......+......+...+..............+............+....+..+...+.........+.+..+...+....+......+.....................+...+...+..+......+.+...+...+.....+..........+...+..............+.+.....+....+...+..............+.........+.+.........+..+.........+......+....+...++++++
.....+.......+..+.......+...+...+..+.......+.....+.........+.......+...+..+....+...........+...+.......+..+.+.........+.....+.............+.........+...+..+.+.....+...+............+...+......+.+...+..+....+...+..+++++++++++++++++++++++++++++++++++++++*..+......+++++++++++++++++++++++++++++++++++++++*....+......+.++++++
-----
Doing Player-Data
Data done
Starting Computation Parties
Running /home/tobias/Git/SAFEFL/mpspdz/Scripts/../mascot-party.x 0 mpc_fedavg_server-14000-3372-30-2000-200-1-1 -pn 19287 -h localhost -N 2
Running /home/tobias/Git/SAFEFL/mpspdz/Scripts/../mascot-party.x 1 mpc_fedavg_server-14000-3372-30-2000-200-1-1 -pn 19287 -h localhost -N 2
Using security parameter 40
Arguments:
Programm: mpc_fedavg_server
Iterations: 2000
Workers: 30
Parameters: 3372
Portnum: 14000
Chunk Size: 200
Threads: 1
Parallel: 1

Start listening on thread 139684085389056
Party 0 is listening on port 14000 for external client connections.
Listening for client connections on base port 14000
Starting aggregation
Thread 139684085389056 found server.
Party 0 received external client connection from client id: 0
Data send
Starting aggregation
Thread 139684085389056 found server.
Party 0 received external client connection from client id: 0
Data send
[...]

This is basically the same output as you got, and it is the expected output. What's happening here is that the main.py starts the aggregation server using the protocol Mascot. You should be able to see two processes named something like mascot-party.x like seen here in the system monitor of Ubuntu:

These processes are running the MPC aggregation server and wait to receive data via a connection on port 14000. Concurrently, the main.py is running the actual training of the local models.

With outputting Listening for client connections on base port 14000 the MPC server says that it is listening on port 14000 to accept connection from a client, in this case main.py. The next line Starting aggregation basically means it is ready to accept a connection.

The line Party 0 received external client connection from client id: 0 means that main.py connected with the MPC server and Data send is output once main.py send all local models to the aggregation server.

Now the MPC server are computing the aggregation by communicating over localhost. The traffic can be seen here as captured by Wireshark:

After the MPC server is done, it will send the aggregation back to main.py, and wait for a new connection by main.py after outputting Starting aggregation to the console.

This output loop will basically continue until the training is finished. As MPC is quite slow, it might take some time for new outputs to appear.

How long have you been waiting after Data send was output to the console? You can check the task manager to see if the Mascot processes are running, and check the network traffic on localhost to see if the processes are communicating with each other. You may simply have to wait longer.

If it is a macOS related issue, I am unfortunately not able to help any further, as I do not have any device running macOS. Maybe some firewall is blocking the communication between the Mascot processes, hindering them from computing the aggregation.

[Quantum-SMC]

Thank you. Do you know if there is something problematic with building MP-SPDZ that is causing this error:
Meanwhile, I built the new version of MP-SPDZ (0.3.9). Because the previous version (0.3.2) that is used in your code required mpir library that has been deprecated.

.../quantum-safefl % poetry run python main.py
cpu
dataset: p0.1_HAR, server_pc: 100, bias: 0.5, nworkers: 30, net: lr, niter: 2000, lr: 0.25, batch_size: 64, nbyz: 6, attack: no, aggregation: fedavg, FLOD_threshold: 0.5, Flame_epsilon: 3000, Flame_delta: 0.001, Number_runs: 1, DnC_niters: 5, DnC_c: 1, DnC_b: 2000, MP-SPDZ: True, Port: 14000, Chunk_size: 200, Protocol: semi2k, Threads: 1, Parallels: 1, Seed: 1, Test Every: 50
LinearRegression(
  (linear): Linear(in_features=561, out_features=6, bias=True)
)
Setting up SSL for 2 parties
...+++++++++++++++++++++++++++++++++++++++*.+..+...+++++++++++++++++++++++++++++++++++++++*..+........+.+...+...........+.......+...+..+.............+.....+............+...+.+.....+................+..+...............+..........+..+.........+......+....+..+....+.....+...+...+.............+.........+..+.+..+.+.....+.......+..............+.+..+.......+.........+............+...+...........+.........+.+...+.....+............+...+.+.....+.+..............+....+..+.+..................+.....+......+....+..+....+.....+...+..........+...+..+.+.....+.+...+..................+.........+......+............+....................+...+.......+......+......+...........+.......+..+.+..............+....+...........+...+...+.........+....+.....+....+..+....+..+.......+......+.....+.+...+...+..+......+........................+.............+.........+.....+....+............+............+........+....+........+.+......+......+..............+...+.+.....+....+...........+...............+.......+..+.+..+....+.........+.....+.......+........+.........................+.....+......++++++
.+...+..+.+..+...+....+.....+......+++++++++++++++++++++++++++++++++++++++*.....+..+.+.....+.......+++++++++++++++++++++++++++++++++++++++*.......+...................................+.+...+.....+................+.....+....+......+...+...+...+..........................+...+.......+......+..+......+....+......+..++++++
-----
...+..+....+.........+..+..........+........+......+............+...+.+...+.....+......+......+.+..+......+.+...+..+.+.....+...+++++++++++++++++++++++++++++++++++++++*.+........+++++++++++++++++++++++++++++++++++++++*..+........+..........+...+.........+......+.........+........+...+.+......+.................+......+.+.........+..+...+............+....+...+..+......+.............+......+.....+....+.....+.+........+...+...++++++
..........+.......+..+.+++++++++++++++++++++++++++++++++++++++*..........+...+....................+.+.....+++++++++++++++++++++++++++++++++++++++*...+....+.................+.......+.....+.+..+..........+..+...+...++++++
-----
Doing Player-Data
Setting up SSL for 1 parties
..+...+......+......+.+...+.....+................+.........+..+...+....+.....+++++++++++++++++++++++++++++++++++++++*..+.....+....+++++++++++++++++++++++++++++++++++++++*....+...+........................+........+....+...+.....+..........+...+...+......+.....+.+...+..+.........+.........+......+....+..+.........+...+..........+..+....+......+........+.......+.....+.+........+............+....+........+................+............+...+..+.......+..+....+..+...+.......+........+....+...........+.+.........+..+....+...+..+.+..+..........+....................+...+....+...+..+................+........+......+.+........+........................+......+....+..+....+.........+..+.+.........++++++
...+......+.........+..+....+++++++++++++++++++++++++++++++++++++++*.......+..+...+.......+...+.....+.+......+...+......+..+++++++++++++++++++++++++++++++++++++++*.+..+....+...+..................+...............+...+.............................+......+......+...+.+............+.....+...............+..........++++++
-----
Doing Player-Data
Data done
Starting Computation Parties
Running /Users/zeinab/Downloads/quantum-safefl/src/quantum-safefl/mpspdz/Scripts/../semi2k-party.x 0 mpc_fedavg_server-14000-3372-30-2000-200-1-1 -pn 16996 -h localhost -N 2
Running /Users/zeinab/Downloads/quantum-safefl/src/quantum-safefl/mpspdz/Scripts/../semi2k-party.x 1 mpc_fedavg_server-14000-3372-30-2000-200-1-1 -pn 16996 -h localhost -N 2
Using statistical security parameter 40
Trying to run 64-bit computation
Arguments:
Programm: mpc_fedavg_server
Iterations: 2000
Workers: 30
Parameters: 3372
Portnum: 14000
Chunk Size: 200
Threads: 1
Parallel: 1

Listening for client connections on base port 14000
Starting aggregation
Traceback (most recent call last):
  File "/Users/zeinab/Downloads/quantum-safefl/src/quantum-safefl/main.py", line 559, in <module>
    main(args)      # call main with parsed arguments
  File "/Users/zeinab/Downloads/quantum-safefl/src/quantum-safefl/main.py", line 474, in main
    aggregation_rules.mpspdz_aggregation(grad_list, net, args.lr, args.nbyz, byz, device, param_num=num_params, port=args.port, chunk_size=args.chunk_size, parties=args.players)
  File "/Users/zeinab/Downloads/quantum-safefl/src/quantum-safefl/aggregation_rules.py", line 476, in mpspdz_aggregation
    output = m.client(0, parties, port, param_num, n, chunk_size, param_list_python, precision=12)
  File "/Users/zeinab/Downloads/quantum-safefl/src/quantum-safefl/mpspdz/ExternalIO/mpc_client.py", line 33, in client
    client.send_private_inputs(input[i * chunk_size:(i + 1)*chunk_size])
  File "/Users/zeinab/Downloads/quantum-safefl/src/quantum-safefl/mpspdz/ExternalIO/client.py", line 58, in send_private_inputs
    triples = self.receive_triples(T, len(values))
  File "/Users/zeinab/Downloads/quantum-safefl/src/quantum-safefl/mpspdz/ExternalIO/client.py", line 52, in receive_triples
    raise Exception(
Exception: invalid triple 0, diff -0x76343d7005a9294e
Fatal error: read_some: stream truncated [asio.ssl.stream:1]
=== Party 1
Using statistical security parameter 40
Trying to run 64-bit computation
Fatal error: read_some: Connection reset by peer [system:54]

Best,

[Tobias512]

I looked into the problem, and it appears that the newer MP-SPDZ versions changed how the multiplication triples are generated. The problem is that in 0.3.2 there was a bug in the client.py file that was causing the triples to be generated incorrectly. You can see our correctly to the file here:

SAFEFL/mpspdz/ExternalIO/client.py

Lines 36 to 38 in 31773ec

	def receive_triples(self, T, n):
	#triples = [[0, 0, 0] * n]
	triples = [[0, 0, 0] for i in range(n)]

The data structure was the wrong format. Therefore, we have this file in the repository to override the existing file present in MP-SPDZ to make the framework work.

This bug in the example was apparently fixed in version 0.3.3 looking at the change log (https://github.com/data61/MP-SPDZ/releases/tag/v0.3.3).

Now the problem is that client.py has the structure as in 0.3.2, but since then, the file has changed quite a lot in 0.3.9 as seen here:
https://github.com/data61/MP-SPDZ/blob/acc49905b429fb9e323ad80d1c984f6fa8b2490f/ExternalIO/client.py

TLDR, things have changed in MP-SPDZ, which make our provided files needed for 0.3.2 no longer valid. Therefore, using a newer version of MP-SPDZ will simply not work.