r(3621) attempt to write read-only file

[94fangming]

I am trying to use parallel to work with a large dataset: generate several new variables using regular expression. For each variable, the command is in the form of

parallel, by(paragroup): gen age_1_plus = ustrregexs(2) if ///
  ustrregexm(jd, "^(.*)([0-9][0-9]+(-|—|~|～|﹣|－|——|————|--|---|----|-——|——-)[0-9][0-9]+(years old))(.*)$")

And I have around 10 commands (for 10 new variables) like this.

When I run each parallel command separately, they all work well. However, when I run the whole do-file at one time, there will always be one command that returns the error:

Moreover, it is not the same command that returns this error. That is, when I run the do-file for the first time, the second parallel command returns this r(3621) error, when I run the do-file for the second time, the eighth parallel command returns this r(3621) error, and if I run the command the third time, it might be that the fifth parallel command returns this error. Therefore, I am not sure what is the reason and how I can deal with it.

Many thanks for any help or comments!

[94fangming]

I was using Stata 14 when I had the previous issues. When I turned to Stata 16, the dofile works well. For some parallel commands, it will still report the following message, but the dofile still continues to execute.

[gvegayon]

Not sure what could be the issue here. Parallel shouldn't change the access to files. Have you try printing the log to see what is going on?

[DavidNMargolis]

Thanks for checking on this. I'm having the same issue (with Stata 15) as the initial post, with the same error messages. The end of one of the log files reads

. local result = _rc

. if (c(rc)) {
. cd "C:\Users\DNM\ParallelTempFiles/"
C:\Users\DNM\ParallelTempFiles
. mata: parallel_write_diagnosis(strofreal(c("rc")),"C:\Users\DNM\ParallelTempFiles/__pllezfoku93y5_finito0001","while setting memory")
. clear
. exit

end of do-file

Any thoughts? And thanks again.

[DavidNMargolis]

Just a quick follow-up. The error occurred using
parallel, prog(prog1 prog2): prog1 a, opt1(b)
and appears to occur when one of the programs passed along generates an error. Unfortunately, the error codes make this a little hard to recognize, so it might be worth (if possible) changing the error codes?

I hope this helps.

[gvegayon]

Were you using a cluster? I have seen a similar issue when dealing with clusters. Sometimes, files I/O fail because of the lack of permissions between machines. Perhaps it would be a good idea to have the user specify where to store the temp files. Right now it is in c(tmpdir). I'll add this to the wishlist.

[DavidNMargolis]

It was indeed on a cluster. I agree that specifying where the tempfiles are stored could help (I execute a "cd" just before the parallel command to be sure). I hadn't thought that file access across machines could be an issue, but you're right that inappropriate permissions could be a problem (although how to specify the temporary files directory could vary with the architecture of the cluster, I imagine, so providing general advice could be hard).

But do you think that there's a way to capture the error codes returned by the programs that are sent to each processor? In this example, prog2, which was called from prog1 (the program executed by "parallel"), generated an error and it took a while to figure out that this was why parallel sent its odd error message.

Thanks for staying on top of this, and for your work in putting together such a useful package!

[gvegayon]

It was indeed on a cluster.

Yes, I have definitely seen this in the past.

But do you think that there's a way to capture the error codes returned by the programs that are sent to each processor? In this example, prog2, which was called from prog1 (the program executed by "parallel"), generated an error and it took a while to figure out that this was why parallel sent its odd error message.

You could try to wrap your code within a capture and write the output (error) to a shared drive, or something like that.

We'll have to figure this one out... although Pull Requests are welcome :).