Input/Output Errors When Reading/Writing Files After Successfully Mounting CephFS Using go-ceph/cephfs

[herenli]

Hello,
I am encountering input/output errors when trying to read and write files after successfully mounting the Ceph File System (CephFS) using the go-ceph/cephfs library.

Context:
I have successfully mounted the CephFS manually, and I can create directories and files without any issues.
However, when using the go-ceph/cephfs interface to perform read and write operations, I encounter input/output errors.

Error Output:
go run -tags luminous test.go
Success creating CephFS mount
Success reading default config file
Success mounting CephFS
Directory created successfully.
Error writing to file: cephfs: ret=5, Input/output error

Details:
go-ceph v0.4.0
ceph luminous

Could anyone provide insights or suggestions on why this might be happening? Any help would be greatly appreciated!
Thank you!

the code is attached.
code.md

[nixpanic]

go-ceph v0.4.0
ceph luminous

Wow, these are both quite old and not maintained anymore.

There is a very similar existing test:

go-ceph/cephfs/file_test.go

Lines 80 to 96 in 2c86010

	t.Run("writeAndRead", func(t *testing.T) {
	// idempotent open for read and write
	f1, err := mount.Open(fname, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
	n, err := f1.Write([]byte("yello world!"))
	assert.NoError(t, err)
	assert.EqualValues(t, 12, n)
	err = f1.Close()
	assert.NoError(t, err)
	defer func() { assert.NoError(t, mount.Unlink(fname)) }()
	buf := make([]byte, 1024)
	f2, err := mount.Open(fname, os.O_RDONLY, 0)
	n, err = f2.Read(buf)
	assert.NoError(t, err)
	assert.EqualValues(t, 12, n)
	assert.Equal(t, "yello world!", string(buf[:n]))
	})

I have added an extra mount.MakeDir() and put the fname in the new directory, like this:

	t.Run("writeAndReadInDir", func(t *testing.T) {
		dirname := "/TestDir"
		dfname := dirname + "/" + fname

		err := mount.MakeDir(dirname, 0777)
		assert.NoError(t, err)
		defer func() { assert.NoError(t, mount.RemoveDir(dirname)) }()

		// idempotent open for read and write
		f1, err := mount.Open(dfname, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
		n, err := f1.Write([]byte("yello world!"))
		assert.NoError(t, err)
		assert.EqualValues(t, 12, n)
		err = f1.Close()
		assert.NoError(t, err)
		defer func() { assert.NoError(t, mount.Unlink(dfname)) }()

		buf := make([]byte, 1024)
		f2, err := mount.Open(dfname, os.O_RDONLY, 0)
		n, err = f2.Read(buf)
		assert.NoError(t, err)
		assert.EqualValues(t, 12, n)
		assert.Equal(t, "yello world!", string(buf[:n]))
	})

After that, running make test-container passes the new TestFileReadWrite/writeAndReadInDir test.

=== RUN   TestFileReadWrite
=== RUN   TestFileReadWrite/writeAndRead
=== RUN   TestFileReadWrite/writeAndReadInDir
=== RUN   TestFileReadWrite/openForWriteOnly
=== RUN   TestFileReadWrite/openForReadOnly
=== RUN   TestFileReadWrite/uninitializedFile
--- PASS: TestFileReadWrite (0.01s)
    --- PASS: TestFileReadWrite/writeAndRead (0.00s)
    --- PASS: TestFileReadWrite/writeAndReadInDir (0.00s)
    --- PASS: TestFileReadWrite/openForWriteOnly (0.00s)
    --- PASS: TestFileReadWrite/openForReadOnly (0.00s)
    --- PASS: TestFileReadWrite/uninitializedFile (0.00s)

This probably indicates a permission or other configuration mistake on the Ceph cluster. Maybe logs from the MDS or OSD processes give a hint on what could be wrong?

Have you used the same credentials for mounting with kernel or fuse CephFS, and did that work?

[phlogistonjohn]

I'll 2nd the "wow that's really old" comment from @nixpanic - it's so old that I hesitate to invest a lot of effort here. Please consider upgrading to something more recent & supported by the upstream projects.

I will add that ceph has various debugging option that can be loaded via a ceph.conf or config set functions. See https://docs.ceph.com/en/latest/rados/troubleshooting/log-and-debug/
The Ceph libraries will then emit debug logging (to the standard error IIRC).

[phlogistonjohn]

Oh, I should add I generally find it simpler to turn them all on to level 10. It produces a huge amount of output but I never seem to be able to guess the proper subsystem name for what I am looking for and so I just prefer to cast a wide net.

[herenli]

@nixpanic Thank you for your reply.
I did a similar existing test, but still have an error

--- FAIL: TestFileReadWrite (0.07s)
    --- FAIL: TestFileReadWrite/writeAndRead (0.01s)
    --- PASS: TestFileReadWrite/openForWriteOnly (0.00s)
    --- PASS: TestFileReadWrite/openForReadOnly (0.00s)
    --- PASS: TestFileReadWrite/uninitializedFile (0.00s)

And I used ceph-fuse to mount the cephfs manually and can read/write the file successfully.
So, I don't know the reasons.

[herenli]

@phlogistonjohn Thank you for your reply.
I updated the go-ceph version to v0.11 for using of nautilus.
But I still have the input/output errors when reading/writing files after successfully mounting cephfs using go-ceph/cephfs.

=== RUN   TestFileReadWrite/writeAndRead
    cephfs_test.go:336: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:336
                Error:          Received unexpected error:
                                cephfs: ret=-5, Input/output error
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:337: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:337
                Error:          Not equal: 
                                expected: 12
                                actual  : 0
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:345: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:345
                Error:          Received unexpected error:
                                cephfs: ret=-5, Input/output error
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:346: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:346
                Error:          Not equal: 
                                expected: 12
                                actual  : 0
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:347: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:347
                Error:          Not equal: 
                                expected: "yello world!"
                                actual  : ""
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -yello world!
                                +
                Test:           TestFileReadWrite/writeAndRead

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contribution.