[WIP] Makefile [WIP] #64

NobbZ · 2016-03-29T12:01:04Z

This PR is created as a WIP to solve #52.

TODO:

juditacs · 2016-03-29T12:24:11Z

FYI I just merged a C version.

NobbZ · 2016-03-29T12:33:49Z

I've seen the notification email.

Also I do a rebase every now and then to keep track of upstream changes.

Of course I'll do a final rebase when it is done, to make it auto-mergeable.

NobbZ · 2016-03-29T12:47:20Z

As I'm still drafting needed targets, what's the purpose of compare_full.sh? I took a look into it's source and it just seems to compare the output of every program against bash's output as a reference, is this right? Can I consider this an extended test?

edit

I tried it, using the data/test/*.in files and everything does fine (except for the cpp_baseline and clang versions, which are not compatible to that old script due to my changes reagrding the Makefile, but this will definetively change as the Makefile improves).

Using data/huwikisource-latest-pages-meta-current.xml does give me the following:

$ scripts/compare_full.sh data/huwikisource-latest-pages-meta-current.xml 
cpp/wordcount output differs from etalon
scripts/compare_full.sh: line 23: cpp/wordcount_clang: No such file or directory
cpp/wordcount_clang output differs from etalon
scripts/compare_full.sh: line 23: cpp/wordcount_baseline: No such file or directory
cpp/wordcount_baseline output differs from etalon
d/wordcount output differs from etalon
elixir/wordcount output differs from etalon
go/bin/wordcount output differs from etalon
haskell/WordCount output differs from etalon
java -classpath java WordCount output differs from etalon
java -cp clojure.jar clojure.main clojure/wordcount.clj output differs from etalon
julia julia/wordcount.jl output differs from etalon
lua lua/wordcount.lua output differs from etalon
mono csharp/WordCountList.exe output differs from etalon
nodejs javascript/wordcount.js output differs from etalon
nodejs typescript/wordcount.js output differs from etalon
perl/wordcount.pl output differs from etalon
php5.6 php/wordcount.php output differs from etalon
php7.0 php/wordcount.php output differs from etalon
python/wordcount_py2_baseline.py output differs from etalon
python/wordcount_py2.py output differs from etalon
python/wordcount_py3.py output differs from etalon
rust/wordcount/wordcount output differs from etalon
scala -J-Xmx2g -classpath scala Wordcount output differs from etalon

juditacs · 2016-03-29T13:01:36Z

compare_full is an old script that takes one output as the etalon and compares the rest with it. Unfortunately they differ so much that it makes little sense to use this script at all. Maybe we should delete it.

NobbZ · 2016-03-29T13:05:48Z

Maybe we should delete it.

Okay, so I will not consider it during the transition, but I will not delete it either. If you really think it is obselete please do that change upstream, it's your file and your project at last.

NobbZ · 2016-03-29T13:34:20Z

There are currently two C# sourcefiles, both capable to be compiled into an exectuable file, but only one of them is used in build.sh.

Shall I drop the other one silently, keep both but compile only the one which is in build.sh, or split it up as I did with C++ and leave them for a later housekeeping?

juditacs · 2016-03-29T14:15:43Z

I think we only want to keep one of them (WordCount), you can ignore the other. There is a pending C# PR, after that, I'll clean up the directory.

NobbZ · 2016-03-29T14:19:39Z

So I'll wait doing the C# related stuff a bit until that got merged.

juditacs · 2016-03-29T14:24:19Z

Thanks, I'll try to speed it up. If the author doesn't respond, I'll finish it.

NobbZ · 2016-03-29T14:29:11Z

Take your time, as you can see in the to-do list, there are quite some
items left which I need time to think about integrating in the Makefile
chain.

Judit Acs [email protected] schrieb am Di., 29. März 2016 16:24:

Thanks, I'll try to speed it up. If the author doesn't respond, I'll
finish it.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#64 (comment)

juditacs · 2016-03-29T14:36:49Z

Makefile.template

@@ -0,0 +1,13 @@
+# For scripted languages it might be unnecessary to change anything except the line below `run:`


I think a "not" is missing here.

Where? There's also a “not” hidden in the word “unnecessary”, but I can change the sentence using “not” directly, which would make it a little bit more clear.

What do you think about:

# For scripted languages it might **not** be necessary to change anything except the line below `run:`

Or is my understanding of the english language fooling me, and both sentences do actually something very different to what I want to say here? English isn't my primary language, only secondary and nearly exclusively used while reading.

Neither is it my native language, which is obvious from my reading mistake :)
It's clear from the context, no need to change anything. I just thought there was a mistake.

juditacs · 2016-03-29T14:40:53Z

Thanks for the amazing work. Let me know if I can help with anything. I also have a contributing colleague, he can help too.

I'm a total make noob, but it looks very good.

If I understand correctly, your goal is to have a single wordcount "binary" in each directory. This is of course a problem for Python and PHP. Python can be split into two directories, they are not really the same language anyway. As for PHP, we can have two directories. One of them stores the source and the other one has only a makefile which "builds" the same wordcount "binary" but runs it with the other PHP version. What do you think?

NobbZ · 2016-03-29T14:47:51Z

Yeah, I've also thought about this. Something like a run rule like php6 ../php5/wordcount.php or whatever the exact call is.

NobbZ · 2016-03-29T23:50:22Z

OK, all languages do have a Makefile now.

make: builds everything
make build_<lang>: build the binary of <lang>
make run: runs everything against huwikisource
make run_<lang>: runs <lang> against huwikisource
make clean: deletes all binaries and buildartifacts
make clean_<lang>: deletes all binaries and buildartifacts of <lang>
make rebuild: rebuilds everything

The output of time during a run is very rudimentary for now, format used is %e__%U__%M as it was by the old runner.
The format used by time during a run can be changed from CLI by providing the variable TIMEFORMAT: make run TIMEFORMAT="%e; %U; %M"
You can change the path to time by providing TIME variable to make
You can change the file that is piped into the testsubjects by providing its path via TESTDATA
Instead of make run_cpp run_elixir you can also use the variable LANGUAGES to selectively build/clean/rebuild/run languages. Make sure that you assign an space separated list of language names enclosed by doublequotes: make clean LANGUAGES="cpp cpp_baseline elixir rust"
Languages may define their own set of variables (as done in elixir) but are required to define proper defaults! [elixir isn't using that variable right now for anything, but it was an experiment]

Currently it is not possible to run a full competition through the Makefile, but for today I'm done and need some sleep ;)

And of course combinations of the variables are possible (times are dirty, I used the system during run!):

$ make run LANGUAGES="cpp cpp_clang rust" TESTDATA=data/huwiki-latest-pages-meta-current.xml 
running "cpp"
163.97__159.68__4373348
finished running "cpp"
running "cpp_clang"
155.50__151.38__4373312
finished running "cpp_clang"
running "rust"
289.07__128.79__6679496
finished running "rust"

NobbZ · 2016-04-07T21:36:46Z

Sorry for beeing silent on this, but I had to slow down a bit.

I will do another rebase tomorrow to keep up with your work. Then I'll start thinking about how to do the remaining parts.

Since my brother in law visits over the weekend, I do think, that I won't actually commit something before monday.

I'm not quite happy with this, since it the time used to compile the file into a java-class is counted in as time processing time in the contest. But I'll leave it as is for the following reasons: * It was this way before considering the `run_commands.txt` * I haven't found a way to do it better without introducing build tools So the “maintainers” of clojure need to chime in here and find a proper way of precompiling it. Or let everything as is, if it is the clojure way. Also I have removed downloading `clojure.jar` from `Dockerfile` and put it in clojures `Makefile`. This way, it is not necessary to rebuilt the image just to update it's version, also the clojure related JAR isn't lingering in the project root anymore ;)

Elixir was one of THE first languages that got a Makefile, I just did forgot to remove it from that script back then.

Both errors slipped through before, because I was in a hurry when doing them and I forgot to test them.

juditacs · 2016-06-14T14:23:59Z

How is this coming along? Is there an easy way to add a "simple/idiomatic" and an "optimized" version? It seems more important than a parallel/single thread switch.

#91

NobbZ force-pushed the makefile branch from 84f6016 to 994972c Compare March 29, 2016 12:07

NobbZ force-pushed the makefile branch from f341329 to 45d8d2c Compare March 29, 2016 14:12

juditacs reviewed Mar 29, 2016
View reviewed changes

NobbZ force-pushed the makefile branch 2 times, most recently from 78513f1 to f8158d4 Compare March 29, 2016 20:20

NobbZ mentioned this pull request Apr 1, 2016

Docker rewrite, cleanup, and enable easy, direct use, and make available via DockerHub #70

Merged

6 tasks

NobbZ added 10 commits April 8, 2016 12:05

Add make to Dockerfile

9862890

First draft of global Makefile

cb2d7ee

Elixir Makefile

33fd16f

Added fallback into Makefile

2f6ddda

Added some status output to global Makefile

214b6e2

Split up cpp and cpp_wordcount into separate folders

5bc8e2f

Makefiles for both CPP examples

20f2da8

Better pattern for ignoring compiled binaries

899209c

Makefile for D

cb47529

[bash] add Makefile

c7dd930

NobbZ added 28 commits April 8, 2016 12:05

[base] add Makefile.template

c1de5d5

[go] add Makefile

7834300

[haskell] add Makefile

ce19357

[java] add Makefile

829dbfb

[javascript] add Makefile

aedf5c0

[julia] add Makefile

7eeaed6

[lua] add Makefile

529da3e

[perl] add Makefile

6d18bfb

[rust] add Makefile

f5e225a

[scala] add Makefile

acf8bd6

[typescript] Makefile

a22853a

[elixir] remove from scripts/build.sh

90e2437

Elixir was one of THE first languages that got a Makefile, I just did forgot to remove it from that script back then.

[elixir] [go] fixed Makefile

f4a5a27

Both errors slipped through before, because I was in a hurry when doing them and I forgot to test them.

[cpp] add Makefile for clang versions

7e39da8

[c] add Makefile

7aa76f3

[csharp] add Makefile

4bb4eed

[base] gitignore update

c588dbd

[base] remove scripts/build.sh

c662a98

[php5.6] prepared folder

fb22e4d

[php5.6] add Makefile

f3be8e8

[php7.0] add Makefile

35fdf39

[python] split

31441f7

[py2_baseline] add Makefile

3354d90

[py2] add Makefile

b645991

[py3] add Makefile

98bc2f1

[base] removed fallback in Makefile

adad903

[base] fetch testdata

11c1931

NobbZ force-pushed the makefile branch from 99d9abf to 11c1931 Compare April 8, 2016 10:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Makefile [WIP] #64

[WIP] Makefile [WIP] #64

NobbZ commented Mar 29, 2016

juditacs commented Mar 29, 2016

NobbZ commented Mar 29, 2016

NobbZ commented Mar 29, 2016

juditacs commented Mar 29, 2016

NobbZ commented Mar 29, 2016

NobbZ commented Mar 29, 2016

juditacs commented Mar 29, 2016

NobbZ commented Mar 29, 2016

juditacs commented Mar 29, 2016

NobbZ commented Mar 29, 2016

juditacs Mar 29, 2016

NobbZ Mar 29, 2016

juditacs Mar 29, 2016

juditacs commented Mar 29, 2016

NobbZ commented Mar 29, 2016

NobbZ commented Mar 29, 2016

NobbZ commented Apr 7, 2016

juditacs commented Jun 14, 2016

		@@ -0,0 +1,13 @@
		# For scripted languages it might be unnecessary to change anything except the line below `run:`

[WIP] Makefile [WIP] #64

Are you sure you want to change the base?

[WIP] Makefile [WIP] #64

Conversation

NobbZ commented Mar 29, 2016

TODO:

juditacs commented Mar 29, 2016

NobbZ commented Mar 29, 2016

NobbZ commented Mar 29, 2016

juditacs commented Mar 29, 2016

NobbZ commented Mar 29, 2016

NobbZ commented Mar 29, 2016

juditacs commented Mar 29, 2016

NobbZ commented Mar 29, 2016

juditacs commented Mar 29, 2016

NobbZ commented Mar 29, 2016

juditacs Mar 29, 2016

Choose a reason for hiding this comment

NobbZ Mar 29, 2016

Choose a reason for hiding this comment

juditacs Mar 29, 2016

Choose a reason for hiding this comment

juditacs commented Mar 29, 2016

NobbZ commented Mar 29, 2016

NobbZ commented Mar 29, 2016

NobbZ commented Apr 7, 2016

juditacs commented Jun 14, 2016