Add ability to retry transactions

[jamesdoherty]

We are seeing exceptions from time to time that are related to transaction commit failure. This looks to be causing issues with certain database fields not being updated when we're expecting them to (eg, the submittime field on the userexams table). OASIS should attempt to retry transactions that fail.

An couple of example exceptions:

Exception on /assess/assessmentpage/3/224/1 [POST] Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1504, in wsgi_app
    response = self.full_dispatch_request()
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1264, in full_dispatch_request
    rv = self.handle_user_exception(e)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1262, in full_dispatch_request
    rv = self.dispatch_request()
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1248, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
File "/opt/oasisqe/3.9/src/oasis/__init__.py", line 107, in call_fn
    return func(*args, **kwargs)
File "/opt/oasisqe/3.9/src/oasis/views_assess.py", line 137, in assess_assessmentpage
    Exams.set_user_status(user_id, exam_id, 2)
File "/opt/oasisqe/3.9/src/oasis/lib/Exams.py", line 167, in set_user_status
    run_sql("""UPDATE userexams SET status=%s WHERE exam=%s AND student=%s;""", (status, exam, student))
File "/opt/oasisqe/3.9/src/oasis/lib/DB.py", line 45, in run_sql
    dbpool.commit(conn)
File "/opt/oasisqe/3.9/src/oasis/lib/Pool.py", line 103, in commit
    dbc.commit()
File "/opt/oasisqe/3.9/src/oasis/lib/Pool.py", line 68, in commit
    self.conn.commit()
TransactionRollbackError: could not serialize access due to read/write dependencies among transactions
DETAIL:  Reason code: Canceled on identification as a pivot, during commit attempt.
HINT:  The transaction might succeed if retried.


Exception on /practice/markquestion/19/1985068 [POST] Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1504, in wsgi_app
    response = self.full_dispatch_request()
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1264, in full_dispatch_request
    rv = self.handle_user_exception(e)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1262, in full_dispatch_request
    rv = self.dispatch_request()
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1248, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
File "/opt/oasisqe/3.9/src/oasis/__init__.py", line 107, in call_fn
    return func(*args, **kwargs)
File "/opt/oasisqe/3.9/src/oasis/views_practice.py", line 271, in practice_mark_question
    marking = Practice.mark_q(user_id, topic_id, question_id, request)
File "/opt/oasisqe/3.9/src/oasis/lib/Practice.py", line 209, in mark_q
    DB.set_q_status(q_id, 3)    # 3 = marked
File "/opt/oasisqe/3.9/src/oasis/lib/DB.py", line 160, in set_q_status
    run_sql("UPDATE questions SET status=%s WHERE question=%s;", (status, q_id))
File "/opt/oasisqe/3.9/src/oasis/lib/DB.py", line 44, in run_sql
    res = conn.run_sql(sql, params, quiet=quiet)
File "/opt/oasisqe/3.9/src/oasis/lib/Pool.py", line 51, in run_sql
    rec = cur.execute(sql, params)
TransactionRollbackError: could not serialize access due to concurrent update

[colincoghill]

Potentially very serious - could lose data if not fixed.

[colincoghill]

Ok, after some experimenting with the database layer, it looks like it'll need quite a lot of work. It's high priority, so will keep on it.

[colincoghill]

Created a dbchange branch. Gonna need to change out most of the database layer. Hopefully can do it
piecemeal.

[colincoghill]

ok working on dbchange in parallel with master, since it's such a big change.

[colincoghill]

Redid the way DB calls are done to hopefully remove the need for explicit transactions.

e736c44

[colincoghill]

Looks pretty good so far, still trying to prove that it works better.