Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Created test domain, but program cannot solve it. #77

Open
Angular-Angel opened this issue May 10, 2021 · 7 comments
Open

Created test domain, but program cannot solve it. #77

Angular-Angel opened this issue May 10, 2021 · 7 comments

Comments

@Angular-Angel
Copy link

Angular-Angel commented May 10, 2021

I decided to try the thing on some fuzzy math, and I got the domain working - but it fails to solve even the simplest problem, requiring the use of only one primitive. Now, I'm running it with the same command as used in the incr.py script, so that may be the issue - but if so, can you tell me what I should look into changing? Also, is there documentation of what all the runtime arguments do? There are a bunch, and I haven't been able to find much explanation on them.

Test Script:

import os
import random

try:
    import binutil  # required to import from dreamcoder modules
except ModuleNotFoundError:
    import bin.binutil  # alt import if called as module

from dreamcoder.ec import commandlineArguments, ecIterator
from dreamcoder.grammar import Grammar
from dreamcoder.program import Primitive
from dreamcoder.task import Task
from dreamcoder.type import arrow, treal
from dreamcoder.utilities import numberOfCPUs

# Primitives
def _fand(x, y): return lambda x, y : x * y
def _for(x, y): return lambda x, y : x + y - x * y

def fAnd():
    x = random.choice(range(0, 100)) / 100
    y = random.choice(range(0, 100)) / 100
    return {"i": (x, y), "o": x * y}

def get_treal_task(item):
    return Task(
        item["name"],
        arrow(treal, treal, treal),
        [((ex["i"],), ex["o"]) for ex in item["examples"]],
    )

if __name__ == "__main__":
	
	args = commandlineArguments(
		enumerationTimeout=10, activation='tanh',
		iterations=10, recognitionTimeout=3600,
		a=3, maximumFrontier=10, topK=2, pseudoCounts=30.0,
		helmholtzRatio=0.5, structurePenalty=1.,
		CPUs=numberOfCPUs())
	
	timestamp = datetime.datetime.now().isoformat()
	outdir = 'experimentOutputs/fuzzy/'
	os.makedirs(outdir, exist_ok=True)
	outprefix = outdir + timestamp
	args.update({"outputPrefix": outprefix})
	
	primitives = [
	    Primitive("fand", arrow(treal, treal, treal), _fand),
	    Primitive("for", arrow(treal, treal, treal), _for),
	]

	grammar = Grammar.uniform(primitives)
	
	training_examples = [
		{"name": "fand", "examples": [fAnd() for _ in range(5000)]},
		#{"name": "for", "examples": [add2() for _ in range(5000)]},
		#{"name": "fand*for", "examples": [add3() for _ in range(5000)]},
	]
	training = [get_treal_task(item) for item in training_examples]

    # Testing data

	testing_examples = [
		{"name": "fand2", "examples": [fAnd() for _ in range(500)]},
	]
	testing = [get_treal_task(item) for item in testing_examples]

	# EC iterate

	generator = ecIterator(grammar,
						   training,
						   testingTasks=testing,
						   **args)
	for i, _ in enumerate(generator):
		print('ecIterator count {}'.format(i))

Run Command:

python bin/fuzzy.py -t 2 --testingTimeout 2

@Angular-Angel
Copy link
Author

And I did of course remember to edit program.ml to add the new primitives. The program runs, it just doesn't solve anything for this domain.

@Angular-Angel
Copy link
Author

Angular-Angel commented May 17, 2021

I think I must have some kind of error that prevents it from coming up with viable programs? Maybe my inputs are formatted wrong or something. Does anyone know how to test just the part where it comes up with programs?

@Angular-Angel
Copy link
Author

Okay, yeah, I'm pretty sure this is cause I don't have it configured to handle multiple inputs.

@madhavk98
Copy link

Hi @Angular-Angel how did you configure it to handle multiple inputs? Thanks!

@Angular-Angel
Copy link
Author

Nope, never did, sorry. :/

@sneiman
Copy link

sneiman commented Mar 19, 2022

Hello all -

I am failing to build the ocaml binaries, so am prevented from building my own domain. I got everything else to work well. I have put up an issue - #96 - describing the problems. Hoping you all have a few minutes to see if you can give me some insight ...

thanks

@DanieleMarchei
Copy link

After a lot of trial and error I manage to make this use case work, here is the python script:

import datetime
import os
import random

try:
    import binutil # required to import from dreamcoder modules
except ModuleNotFoundError:
    import bin.binutil # alt import if called as module

from dreamcoder.ec import commandlineArguments, ecIterator
from dreamcoder.grammar import Grammar
from dreamcoder.program import Primitive
from dreamcoder.task import Task
from dreamcoder.type import arrow, tlist, treal
from dreamcoder.utilities import numberOfCPUs


# These four definitions work the same... for some reason

def _fand(x): return lambda y : x * y
def _for(x): return lambda y : x + y - x * y

# def _fand(x, y): return x * y
# def _for(x, y): return x + y - x * y

# def _fand(x, y): return 3
# def _for(x, y): return 5

# def _fand(x, y): return lambda x, y :  x * y
# def _for(x, y): return lambda x, y :  x + y - x * y




def fAnd():
    x = random.choice(range(0, 100)) / 100
    y = random.choice(range(0, 100)) / 100
    
    # we return the two inputs separately and not in a tuple
    return {"x":x, "y":y, "o": x * y}

def get_treal_task(item):
    return Task(
        item["name"],
        arrow(treal, treal, treal),

        # we have two inputs, so we have to provide something in the form ((input1, input2), output)
        [((ex["x"],ex["y"]), ex["o"]) for ex in item["examples"]],
    )

if __name__ == "__main__":

    args = commandlineArguments(
        enumerationTimeout=10, activation='tanh',
        iterations=10, recognitionTimeout=3600,
        a=3, maximumFrontier=10, topK=2, pseudoCounts=30.0,
        helmholtzRatio=0.5, structurePenalty=1.,
        CPUs=numberOfCPUs())

    timestamp = datetime.datetime.now().isoformat()
    outdir = 'experimentOutputs/fuzzy/'
    os.makedirs(outdir, exist_ok=True)
    outprefix = outdir + timestamp
    args.update({"outputPrefix": outprefix})

    primitives = [
        Primitive("fand", arrow(treal, treal, treal), _fand),
        Primitive("for", arrow(treal, treal, treal), _for),
    ]

    grammar = Grammar.uniform(primitives)

    training_examples = [
        {"name": "fand", "examples": [fAnd() for _ in range(5000)]},
    ]
    training = [get_treal_task(item) for item in training_examples]

    # Testing data

    testing_examples = [
        {"name": "fand2", "examples": [fAnd() for _ in range(500)]},
    ]
    testing = [get_treal_task(item) for item in testing_examples]

    # EC iterate

    generator = ecIterator(grammar,
                        training,
                        testingTasks=testing,
                        **args)
    for i, _ in enumerate(generator):
        print('ecIterator count {}'.format(i))

And here are the two primitives added to the program.ml file:

let primitive_fand = primitive "fand" (treal @> treal @> treal) (fun x y -> x *. y);;
let primitive_for = primitive "for" (treal @> treal @> treal) (fun x y -> x +. y -. x *. y);;

Be careful when operating with floating numbers in OCaml because, for example, if you need to perform addition then you need to use the +. operator instead of the regular +. Same thing for all the other operators.

However this does not solve the fundamental issue here: how do the python primitives have to be defined? As you can see by the above script, there are some very funky primitives and all of them work if we use the ocaml solver. If we switch to the python solver then only the first one works.
I think this gives us a clue on how to define primitives with multiple inputs. In fact, the script dreamcoder/domains/list/listPrimitives.py uses this exact semantic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants