Throws an error when trying to open file inside goroutine

[ghost]

Thanks for this nice little module. I am having trouble when trying to run this code with a go subroutine.

The below code works perfectly without go subroutine in the anonymous function inside for loop. But fails with error http request error: <URL> connect: can't assign requested address hen tried with go subroutine.

func main() {
	req, _ := http.NewRequest("GET", "https://dl.google.com/go/go1.10.windows-amd64.zip", nil)

	htrdr, err := httpreaderat.New(nil, req, nil)
	if err != nil {
		panic(err)
	}
	bhtrdr := bufra.NewBufReaderAt(htrdr, 1024*1024)

	zrdr, err := zip.NewReader(bhtrdr, htrdr.Size())
	if err != nil {
		panic(err)
	}

       	var wg sync.WaitGroup
	for _, f := range zrdr.File {
                fileToCopy := f
		go func(fileToCopy *zip.File) {
                    fr, err := f.Open()
		    check(err)
		    fileloc := f.FileInfo().Name()))
                     
                     outFile, err := os.Create(fileloc)
		    check(err)
		    defer outFile.Close()
				
                     _, err = io.Copy(outFile, fr)
		    if err != nil {
		        log.Fatalf("Error: %v", err)
		    }
		    fr.Close()
		    wg.Done()
                  }(fileToCopy)
	}
       wg.Wait()
}

Have I missed the mark anywhere

[snabb]

Most likely you are hitting some kind of limit when you try to do 9252 parallel operations. For example the default number of maximum open files on Linux is usually 1024.

[ghost]

thanks @snabb, the max open file is something I haven't thought about 👏👏.

I tried to limit the max concurrency to only 2 at a time using goccm but it fails after some time with error Error: flate: corrupt input before offset 3393. If I set the limit to 1 it does run successfully. So it may be moving the file pointer too quickly and reaching the EOF.

        c := goccm.New(2) // runs successfully with 1
	for _, f := range zrdr.File {
	        c.Wait()
                fileToCopy := f
		
                go func(fileToCopy *zip.File, ccm goccm.ConcurrencyManager) {
                      // downloadFunc
                      ccm.Done()
                }(fileToCopy, c)
	}
       c.WaitAllDone()```

For now, I am assuming that its not safe to run this in parallel due to the limits.