Creating tables in Database docker containers is slow (M1 Preview) #5236
Comments
jamielsharief
changed the title
|
@jamielsharief Have you done the same test on an Intel Mac? I would be interested to know how much slower it is. |
|
Yes I have, it is significantly slower than a Mac Book 2014 8GB ram. |
|
To give you an idea when running PHP unit for a package, it took 3-4 minutes maybe on intel. On this m1 I had to stop it after 15 minutes. The unit test is dropping and recreating tables for each test. Running this query only through TablePlus CREATE TABLE `bookmarks` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`user_id` INT(11) NOT NULL,
`title` VARCHAR(50) NOT NULL,
`description` TEXT,
`url` TEXT,
`category` VARCHAR(80),
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
PRIMARY KEY (id),
INDEX `user_id` (user_id)
) ENGINE = InnoDBMySQL - m1: 655 ms |
|
Interesting, thank you. |
|
Maybe related: evansm7/vftool#14 |
|
Whilst other things can be faster, I have also noticed in Dockerfile |
|
Hello, Just to confirm we have replicated slow table creation/dropping on M1 Docker preview using mariadb docker image. It seems to be at least 4x to 5x slower. I have access to both a 16" MBP and a 13" M1 MBP that I can run any tests/diagnosis needed. Most likely I/O based |
|
Before Docker guys add |
|
@gyf304 thanks, but I am not interested in a hack, I am reporting a possible problem with the preview version. |
The only people who can actually fix this is Docker or Apple. I'm not affiliated with Docker or Apple so I cannot fix it on their behalf. I'm offering a workaround which might not interest you in particular, but may help other people. |
|
@gyf304 You are right sorry. changing kernel configuration in a docker configuration, that sounds like its going be more than a one liner, do you have. link to resource? |
|
Sorry @gyf304 it was late last night here, and I thought you were talking about trying to modify the kernel on my system to get it to work. If it is a something that can be adjusted in docker or the docker image, then it would be very helpful to many developers. |
|
The kernel argument is for the Linux VM started by Docker Desktop. You can probably modify the cmdline in the package (right-click, show package content) |
|
I also noticed that running tests (w/ database tables cleaned using DELETE statements between each test) against PostgreSQL image (postgres:11) is very slow with the Docker Preview on M1 MacBook Pro compared to running the tests against locally installed PostgreSQL, or even against the stable version running on Intel platform. Perhaps this is similar issue than this? Some timings: M1 MacBook Pro running the test suite against local PostgreSQL: 47 seconds Sorry that I can't provide any more details about the test setup. |
|
Updated to 3.1.0 (60984) and still experiencing the slowness. Unfortunately lowering the CPU count in the settings to 1 did not help for me. |
|
Just a reminder on my comment above, whilst this is noticeable on Database systems, also experienced the same with running file based commands in the Dockerfile. e.g |
|
I downloaded the latest Docker M1 Preview today which says it fixes the filesystem slowness, I am not sure but the databases such as MySQL and Maria still struggle at creating tables.
|
|
mysql is an Intel container, isn't it? That's always going to be slow on an M1 machine because the architecture has to be emulated by qemu. |
|
It seems like the latest version added |
|
On the 11th Jan i published a quick benchmark test (above) Today I used table plus on the most recent version to execute this statement and it gives the execution time.
As regards to the ins and outs of MySQL emulation, I cannot answer, I have built the image locally on my M1 and used buildx on remote server, inboth cases MySQL is considerably slower than MariaDB official images. |
|
I just ran a build and the
|
|
Happens with mongodb too. Consistently taking 600-700ms instead of a few ms. |
|
@danielgindi did you try on the Docker release from last Thursday (preview does not update, you need to reinstall) and are you using the official image of MongoDB? |
|
Hello, I have access to a 16" Intel Macbook Pro and a 13" M1 currently with the first preview release of docker, Are we able to collectively come up with a few different tests I can do on both and then upgrade the M1 to the latest preview released last Thursday and essentially provide the results for all 3 variations? I was thinking of the following:
Do you guys/girls have any other suggestions that would help? I'll do this over the weekend and report back.. |
|
@navidnadali I think use MariaDb since it has an official image and the creating tables is still very slow. What about network traffic speed? |
|
I think we can expect very similar results between MariaDb and MySQL based on the nature of the issue but happy to test on that image as well. Ideally I'd like to limit it to the purpose of the issue that is slow I/O on the new M1 docker preview (I've not noticed any negative network performance so far). |
|
@navidnadali From my original test for creating a table (as described above), MySQL took twice as long. I just mentioned MariaDB (i dont use it) because twice its been mentioned that there is no official image. MySQL - m1: 655 ms Here is my docker image for MySQL https://hub.docker.com/repository/docker/jamielsharief/mysql source code also available on GitHub, its my first venture in compiling for different architectures. I think i got it right. |
|
I can confirm the slowness is observable in all of mysql/mariadb/postgres via docker. From what I saw, the problematic queries are creating tables and delete/truncate queries (especially Ended up using homebrew to install the dependencies (not great as it is pretty much impossible to run mysql and mariadb via brew at the same time). (I am on the latest docker preview, BigSur 11.2.1 (20D74), MBP M1 16gb) edit: Also tried to play with number of CPUs and resources, same result. Even if I don't use explicit volumes and keep the defaults, or even when using |
@jamielsharief Yes and yes |
|
I did some measurements as well and tried to dig a little deeper to see what is actually going on. I started with running Locally installed version: Trough docker: I tried to identify interesting metrics, mostly by searching for related issues and experimenting with different trace events. I have no idea if the things that looked interesting to me are actually relevant or if my interpretation is correct by any means 😄 I created a simple program that just opens a file with Digging deeper ( So moving one layer deeper: Running As I don't run any parallel writes this is probably a good measure of the latency between the block driver and the hypervisor. If I try something that does not fsync immediatly: This looks quite good. Especially compared to results from before mitigating the interrupt drop issue (ubuntu vm with irqbalance and no special cmdline): So significant improvements already 😄 . Looking at I verified that this is equal to a VM running directly on |
|
@jnugh I have installed ubuntu server through Parallels Desktop M1 Preview, and setup LXD, installed MySQL, MariaDB etc in containers, and the performance is great. So, I think it is a docker issue. |
|
@jamielsharief I just installed Parallels. It uses a custom hypervisor. |
|
@jnugh Sorry i must of missed the last line of your post, where you said, if I understand correctly you built a VM using the Apple virtualzation framework, and experienced the same thing. If that is the case, that is very interesting indeed. |
|
Exactly. After testing in a docker VM shell, I used evansm7/vftool which is a cli wrapper for the virtualization API. |
|
When the most recent preview version came out I tried to do some benchmarking on it specifically related to seeing slow IO performance with MariaDB. I posted it on here, in case it would be interesting to reproduce: docker/roadmap#142 (comment) |
|
Awesome thanks for the reference 😄 docker/roadmap#142 (comment) (It's probably something that needs to be fixed by Apple) was also my very uneducated (as I have no idea how hypervisors usually handle disk IO) conclusion after doing some additional debugging using the XCode profiler on the new hypervisor. It looked very inefficient in case of sync writes because even if the guest was only writing to disk it did more reads than writes on the image file. Maybe performance could be improved by disabling the journal and switching the database to not buffered writes which is probably okay for a dev environment. |
|
I have a Dockerfile which uses a setup script This is is the start of the setup script It is just so painfully slow, more than 10 minutes, can somebody from docker address what is the situation here, is this is a docker problem or apple problem as suggested by @jnugh ? |
|
Has someone filed a bug with Apple? |
|
Yes, we (Docker) have filed a ticket with Apple and indeed have had a call with their developers about it. We're also wondering if we can speed it up by flushing less often: there would be a slight risk of lost data if there was a power cut or something, but in a development context that might be an acceptable trade-off. |
|
See also #5389. |
|
@stephen-turner Awesome - would it actually be possible for docker to prevent syncs? Maybe by patching the vm kernel. I just tried to monkeypatch fsync in a test 😸 @jamielsharief I tested with your commands. Building a shard object that routes fsync calls to basically no-ops. (Took the code from here: https://www.mjr19.org.uk/IT/apt-get_fsync.html) I built the shared object in a different container so I did not have any additional packages installed in the second run (some packages would have been installed as a dependency of Default (probably what you are seeing, cancelled at some point): Prevent fsyncs (finished): Fast track to reproduce: add this to your Dockerfile, be aware that it actually loads the no-op fsync all the time: Alternative only for your script: If you don't trust to load random so files form the internet: docker run -it ubuntu:20.04 bash
apt update
apt install -y build-essential
# get the c source into the container (e.g. by using docker cp or an editor like vim)
gcc -c -fPIC nosync.c
gcc -shared -o libnosync.so nosync.o
# Copy the so file from the container and use for other projectsC source (copied from the link): #include <fcntl.h>
int fsync(int fd){
if (fcntl(fd,F_GETFL)==-1) return 1; /* Will set errno appropriately */
return 0;
}Keep in mind that his only changes fsync behavior. There seem to be other solutions like this: https://www.flamingspork.com/projects/libeatmydata/ which claims to also patch e.g. |
|
@jnugh Thats amazing, ty for spending the time on that. This particular docker image, which takes 722 seconds to run the setup script, is for an open source project, so i cant really patch it with this wizardary ;) |
|
@jnugh I just checked the advanced settings of my VMs in parallels, and it seems to be using the Apple virtualisation under the hood, which does not make sense, since it does not experience the same the problem.
|
|
There are two different Hypervisor APIs https://developer.apple.com/documentation/hypervisor (probably used by parallels?) and https://developer.apple.com/documentation/virtualization (used by docker). I don't know how much content is shared between them. When I ran Parallels to test. I was not seeing the the process from Virtualization Framework but I also did not check if there were multiple hypervisors available. So if Apple was the default selection I'm pretty sure that it does not use the Virtualization framework which is used by Docker or vftool 🤔 |
|
@jnugh Fair enough, I heard the virtualisation was a new feature, and assumed it was the same thing. |
|
Until this issue is fixed, I'm just running my database outside of the container; https://dev.to/perry/fix-slow-docker-databases-for-apple-silicon-m1-chips-2ip1 Not a perfect solution, but at least I can work with big databases in the meantime :) |
|
Do you have any solution about this performance issue ? (Except using a database outside of the container ^^) |
|
I setup LXC containers for MySQL, MariaDB and Postgres |
|
On the newly released RC2 version using qemu (https://docs.docker.com/docker-for-mac/apple-m1/): 100-200x speedup in the postgres fsync benchmarks compared to the previous release with virtualization framework 👍🏽 |
|
Similar numbers for me on MBA M1, and projects using DBs feels faster too! |
|
Much much much better on an M1 machine 🎉 I have a NodeJS project that utilizes Prisma 2 and I use their "seed" feature as a means of populating my local database. Prior to this change to Docker, that process, for me, took about five minutes. Now it takes 50 seconds 😮 |
|
Originally when i reported the problem , it took 655ms to run the following MySQL statement, I can now confirm that in the latest version (a few days ago), this now is only 35ms. Thanks. |
|
Closed issues are locked after 30 days of inactivity. If you have found a problem that seems similar to this, please open a new issue. Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. |
I have noticed that when running a database server MySQL or Maria DB in a docker container on the M1 Technical Preview, it is extremely slow.
Watching a visual inspection of the table creation you can see how slow it is.
The text was updated successfully, but these errors were encountered: