Issue 32: Update master branch with contents of dev branch #33

RaulGracia · 2020-06-22T14:22:22Z

Change log description
This PR contains the changes from dev branch to be merged on master to reproduce the experiments to be published:

Added new driver for Pravega.
Added transactional writer to Kafka.
Improvement on metrics reporting.

Still, we need to remove all the non-necessary changes to make this fork as close as possible from the official repo. We also need to remove any mention to personal repos and create the necessary images in Pravega dockerhub.

) Import of current code to development branch with all work done so far. Signed-off-by: Claudio Fahey <[email protected]>

Added transaction support for Pravega driver in OpenMessaging Benchmark. Signed-off-by: wenqimou <[email protected]>

Added transaction support for Kafka. Signed-off-by: wenqimou <[email protected]>

Remove P3 Test Driver framework from OpenMessaging Benchmark fork.

…nnection pooling for writers and time windows for readers. Signed-off-by: Raúl Gracia <[email protected]>

Signed-off-by: Raúl Gracia <[email protected]>

Issue 11: Enable stream auto-scaling

Refractor README

Collocate Segment Store and Bookies and set proper config changes. Signed-off-by: wenqimou <[email protected]>

…default_b_dev Fixing issue 18: PravegaConfig.enableTransaction has the default value of true

Increase sleep time to avoid a race condition when a test completes

* Add option to create scope * Add keycloak dependency

) Avoid allocating a new byte buffer for each write for efficiency reasons. Signed-off-by: Raúl Gracia <[email protected]>

Add configuration to enable Tiering in Pulsar. Signed-off-by: Raúl Gracia <[email protected]> Co-authored-by: Malygina <[email protected]>

Signed-off-by: Raúl Gracia <[email protected]>

…e types. Signed-off-by: Raúl Gracia <[email protected]>

Signed-off-by: Raúl Gracia <[email protected]>

.gitignore

benchmark-framework/src/main/java/io/openmessaging/benchmark/WorkloadGenerator.java

benchmark-framework/src/main/java/io/openmessaging/benchmark/worker/LocalWorker.java

claudiofahey · 2020-06-29T19:25:07Z

benchmark-framework/src/main/java/io/openmessaging/benchmark/worker/LocalWorker.java

-    private final ExecutorService executor = Executors.newCachedThreadPool(new DefaultThreadFactory("local-worker"));
+//    private final ExecutorService executor = Executors.newCachedThreadPool(new DefaultThreadFactory("local-worker"));
+//    private final ExecutorService executor = Executors.newFixedThreadPool(16);
+    private final ExecutorService executor = new ForkJoinPool();


We should explain why we changed from newCachedThreadPool to ForkJoinPool. I think I copied this from Pravega Benchmark.

Ok, but what was the reason this change was introduced? The fact that Pravega Benchmark did this does not seem a sufficiently strong reason, right? Is this providing any advantage over the existing thread pool?

Unfortunately, I don't remember exactly why I started using ForkJoinPool. I was likely investigating performance differences between OMB and Pravega Benchmark and I found this to work well. I agree that we should better justify the change or revert it. Are you able to run a benchmark with the original executor?

I will need to do that to see if there is any difference.

bin/benchmark-worker

driver-pravega/doc/build_pravega.md

...-pravega/src/main/java/io/openmessaging/benchmark/driver/pravega/PravegaBenchmarkDriver.java

driver-pulsar/deploy/deploy.yaml

pom.xml

workloads/tiny.yaml

Signed-off-by: Raúl Gracia <[email protected]>

…a/pravega/releases/download/v{{ pravegaVersion}}/pravega-{{ pravegaVersion }}.tgz

jingerbread · 2020-07-08T10:31:07Z

[updated] claudiofahey/pravega-tester:0.7.0 from 0.5.1 version
[changed] pravegaSrcRemote: yes pravegaSrc: https://github.com/pravega/pravega/releases/download...

RaulGracia · 2020-07-13T13:35:55Z

@jingerbread @claudiofahey I don't think that is a good idea to rely on images from personal repos. If we need an image with whatever is in pravega-tester, then we could consider creating one in the official Pravega repo.
Also @jingerbread, the change on pravegaSrcRemote is ok but will take effect once we release Pravega 0.8.

claudiofahey · 2020-07-13T22:49:20Z

@jingerbread @claudiofahey I don't think that is a good idea to rely on images from personal repos. If we need an image with whatever is in pravega-tester, then we could consider creating one in the official Pravega repo.

It will take some time to get approvals to make pravega-tester official. Let's remove it from the Ansible script for now. It is not too important. Pravega-tester just performs a few writes and reads from Pravega as a sanity check.

jingerbread · 2020-07-15T09:10:27Z

@RaulGracia, currently in the PR pravegaVersion: "0.7.1" which is released already and using pravegaSrcRemote https://github.com/pravega/pravega/releases/download/v0.7.1/pravega-0.7.1.tgz will work

RaulGracia · 2020-07-15T09:11:20Z

@jingerbread why have you closed this PR that is not yours and is still under review?

jingerbread · 2020-07-15T09:12:34Z

@jingerbread why have you closed this PR that is not yours and is still under review?

Sorry, I did it by mistake

RaulGracia · 2020-07-15T09:22:06Z

@claudiofahey @jingerbread we can leave at the moment the Pravega tester image as it is, given that it is useful to know if the deployment works. We can address updating that image on a separate issue: #35.

jingerbread · 2020-07-15T16:10:53Z

@RaulGracia, should
export TF_STATE=./ to

be added before
ansible-playbook --user ec2-user --inventory `which terraform-inventory` deploy.yaml

In README.md for https://github.com/pravega/openmessaging-benchmark/tree/issue-32-cleanup-dev/driver-pravega#running-the-ansible-playbook

Now it's in separate Troubleshooting page https://github.com/pravega/openmessaging-benchmark/blob/issue-32-cleanup-dev/driver-pravega/doc/troubleshooting.md

But for proposed in README.md terraform and terraform-inventory versions
ansible-playbook --user ec2-user --inventory `which terraform-inventory` deploy.yaml
will do nothing, because can't get hosts from inventory

[root@W10J6T4733 deploy]# ansible-playbook  --user ec2-user --inventory `which terraform-inventory` deploy.yaml
 [WARNING]:  * Failed to parse /home/aws/tools/terraform-inventory with script plugin: Inventory script (/home/aws/tools/terraform-inventory) had an execution error: Error reading tfstate file: 0.12 format error: <nil>; pre-0.12 format error: <nil> (nil error means no content/modules found in the respective format)

jingerbread · 2020-07-15T16:26:09Z

Should io.pravega:pravega-client version in driver-pravega/pom.xml changed from 0.6.2 version to 0.7.1?

        <dependency>
            <groupId>io.pravega</groupId>
            <artifactId>pravega-client</artifactId>
            <version>0.6.2</version>
        </dependency>

To

<!-- https://mvnrepository.com/artifact/io.pravega/pravega-client -->
<dependency>
    <groupId>io.pravega</groupId>
    <artifactId>pravega-client</artifactId>
    <version>0.7.1</version>
</dependency>

jingerbread · 2020-07-15T16:33:57Z

For Apache Maven 3.6.3

mvn clean install -DskipTests=true

failed on driver-pulsar due to

[ERROR] Failed to execute goal com.mycila:license-maven-plugin:3.0:check (default) on project driver-pulsar: 
Some files do not have the expected license header -> [Help 1]

Need to

 mvn clean install -DskipTests=true -Dlicense.skip=true

jingerbread · 2020-07-15T19:01:19Z

Seems, when Pravega 0.8 will be release, need to change controller.zk.url to controller.zk.connect.uri in templates/controller.config.properties. Change also zookeeper version in vars.yaml to 3.6.1.

May be separate task to update this PR for 0.8 can be created?

[root@W10J6T4733 deploy]# grep -r "controller.zk.url" *
templates/controller.config.properties:controller.zk.url={{ zookeeperServers }}

pravega/pravega@b4cd667
Issue 4900: Fix configuration related issues in templates and docs

...
- "value": "SERVER_OPTS=\"-Dcontroller.zk.url=master.mesos:2181\" ...
+ "value": "SERVER_OPTS=\"-Dcontroller.zk.connect.uri=master.mesos:2181\ ...

Otherwise, now with latest Pravega commit b4cd667 from 15 Jul - Pravega deployment wont't work.
Controller will try to connect to zk on localhost:

2020-07-15 18:13:33,972 319  [main] INFO  io.pravega.controller.util.Config - controller.zk.url = 10.0.0.152:2181,10.0.0.86:2181,10.0.0.105:2181
...
2020-07-15 18:13:34,324 671  [ControllerServiceMain-SendThread(localhost:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused
2020-07-15 18:13:34,327 674  [ControllerServiceMain] INFO  i.p.c.server.ControllerServiceMain - Awaiting ZK client connection to ZK server
2020-07-15 18:13:35,426 1773 [ControllerServiceMain-SendThread(localhost:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-07-15 18:13:35,427 1774 [ControllerServiceMain-SendThread(localhost:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket error occurred: localhost/127.0.0.1:2181: Connection refused

However, current setting should work with Pravega 0.7.1, which is currently referenced in vars.yaml

jingerbread · 2020-07-15T19:48:46Z

With mentioned changes - deployment was successful for latest Pravega commit b4cd667 from 15 Jul

PLAY [Run Pravega Tester] *****************************************************************************************************************************************************************************************************************************************************************************************************

....
PLAY RECAP ********************************************************************************************************************************************************************************************************************************************************************************************************************34.214.51.197              : ok=45   changed=14   unreachable=0    failed=0    skipped=18   rescued=0    ignored=2
34.216.225.48              : ok=34   changed=11   unreachable=0    failed=0    skipped=18   rescued=0    ignored=2
34.217.32.131              : ok=41   changed=15   unreachable=0    failed=0    skipped=16   rescued=0    ignored=2
34.219.134.75              : ok=35   changed=11   unreachable=0    failed=0    skipped=16   rescued=0    ignored=2
34.219.162.179             : ok=41   changed=15   unreachable=0    failed=0    skipped=16   rescued=0    ignored=2
34.219.171.194             : ok=52   changed=23   unreachable=0    failed=0    skipped=15   rescued=0    ignored=1
34.220.119.134             : ok=50   changed=21   unreachable=0    failed=0    skipped=17   rescued=0    ignored=1
52.38.10.6                 : ok=34   changed=11   unreachable=0    failed=0    skipped=18   rescued=0    ignored=2
52.39.178.9                : ok=50   changed=21   unreachable=0    failed=0    skipped=17   rescued=0    ignored=1
54.201.55.191              : ok=36   changed=13   unreachable=0    failed=0    skipped=16   rescued=0    ignored=2
localhost                  : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

RaulGracia · 2020-07-16T10:50:04Z

@jingerbread let me answer your comments:

Agree, export TF_STATE=./ seems a must rather than a corner case to be added in the troubleshooting guide.
Regarding io.pravega:pravega-client version, it should actually be 0.8.0 when it is released, but we can use 0.7.1 at the moment to merge this PR. We will update the version later on.
Regarding -Dlicense.skip=true, if it works also with earlier version of Maven, I'm fine with it too.
Right, we need to change controller.zk.connect.uri for 0.8.0. But this should be done in a separate PR when we upgrade the Pravega artifacts to 0.8.0. Can you file an issue to track that?
Regarding the Zookeeper version, we can leave the one we have, is not important for performance and the current version works well.

@jingerbread as you have these changes in place already, can you commit them to this PR please?

jingerbread · 2020-07-16T11:39:45Z

@RaulGracia

Sure, will commit and create separate ticket for 0.8 changes by the MSK end of the day.

jingerbread · 2020-07-16T18:35:47Z

-Dlicense.skip=true is the com.mycila plugin flag to skip checks, see "Request for a skip check goal. #28" mathieucarbou/license-maven-plugin#28 closed in 2013, so think it will work.
In omb com.mycila.license-maven-plugin:3.0 is used for checking license headers:

<plugin>
	<groupId>com.mycila</groupId>
	<artifactId>license-maven-plugin</artifactId>
	<version>3.0</version>
...

Besides, may be we can fix warnings by adding license headers?

[WARNING] Missing header in: C:/DELL_EMC/Development/OMB/pravega-omb/driver-pulsar/deploy/templates/chrony.conf
[WARNING] Missing header in: C:/DELL_EMC/Development/OMB/pravega-omb/driver-kafka/deploy/templates/chrony.conf
[WARNING] Missing header in: C:/DELL_EMC/Development/OMB/pravega-omb/driver-pravega/deploy/templates/bkenv.sh
...

jingerbread · 2020-07-16T19:25:26Z

Created Update driver-pravega after 0.8.0 Pravega release #36 issue
#36

[changed] driver-pravega/README.md: [added] export TF_STATE=./ before ansible deploy step [changed] driver-pravega/doc/build_pravega.md [added] pravegaSrcRemote and pravegaSrc changing for local artifacts comment [added] comment about -Dlicense.skip=true

RaulGracia · 2020-07-17T08:21:31Z

@jingerbread @claudiofahey I think we can merge this one as the remaining points can be addressed as separate PRs against master branch. What do you think?

jingerbread · 2020-07-17T09:52:20Z

Seem working for 0.7.1, I thing can be merged. Didn't find how to approve this PR.
Added comment about adding license headers to Update driver-pravega after 0.8.0 Pravega release #36 issue

PLAY [Run Pravega Tester] *****************************************************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************************************************************************************************************************************************************ok: [34.212.167.25]

...
PLAY RECAP ********************************************************************************************************************************************************************************************************************************************************************************************************************18.237.21.18               : ok=39   changed=20   unreachable=0    failed=0    skipped=17   rescued=0    ignored=2
34.212.167.25              : ok=48   changed=27   unreachable=0    failed=0    skipped=13   rescued=0    ignored=2
34.217.56.196              : ok=56   changed=32   unreachable=0    failed=0    skipped=12   rescued=0    ignored=3
34.222.245.205             : ok=40   changed=22   unreachable=0    failed=0    skipped=13   rescued=0    ignored=2
52.11.157.73               : ok=39   changed=20   unreachable=0    failed=0    skipped=13   rescued=0    ignored=2
52.12.159.16               : ok=38   changed=20   unreachable=0    failed=0    skipped=15   rescued=0    ignored=2
52.12.248.39               : ok=45   changed=25   unreachable=0    failed=0    skipped=13   rescued=0    ignored=2
54.149.212.20              : ok=45   changed=25   unreachable=0    failed=0    skipped=13   rescued=0    ignored=2
54.213.211.242             : ok=54   changed=30   unreachable=0    failed=0    skipped=14   rescued=0    ignored=3
54.213.57.95               : ok=54   changed=30   unreachable=0    failed=0    skipped=14   rescued=0    ignored=3
localhost                  : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


real    22m2.937s

Claudio Fahey and others added 18 commits November 14, 2019 20:32

Pravega driver, Pravega deployment, framework fixes, test automation (#1

b825799

) Import of current code to development branch with all work done so far. Signed-off-by: Claudio Fahey <[email protected]>

Issue 3: Add transaction support for pravega driver (#2)

958a64b

Added transaction support for Pravega driver in OpenMessaging Benchmark. Signed-off-by: wenqimou <[email protected]>

Issue 6: Add transaction support and script for Kafka (#7)

92800ff

Added transaction support for Kafka. Signed-off-by: wenqimou <[email protected]>

Issue 5: Remove P3 Test Driver framework (#10)

68303e1

Remove P3 Test Driver framework from OpenMessaging Benchmark fork.

Issue 11: Add support for auto-scaling streams. Disable by default co…

6f27e85

…nnection pooling for writers and time windows for readers. Signed-off-by: Raúl Gracia <[email protected]>

Issue 11: Fixed data rate scaling naming and restored Zookeeper version.

1b8c51e

Signed-off-by: Raúl Gracia <[email protected]>

Enable stream auto-scaling

eed4267

Issue 11: Enable stream auto-scaling

Refractor README (#8)

390da0b

Refractor README

Colocating and config changes (#15)

570e109

Collocate Segment Store and Bookies and set proper config changes. Signed-off-by: wenqimou <[email protected]>

BK version and format changes (#16)

0d206a2

Minor change for colocating BK and Segmentstore

47379b5

Increase sleep time to avoid a race condition when a test completes

3d7d1f1

BUG-18 [fixed] PravegaConfig.enableTransaction = false by default

4b2c3ee

Merge pull request #22 from pravega/BUG-18_enableTransaction_true_by_…

8e014ac

…default_b_dev Fixing issue 18: PravegaConfig.enableTransaction has the default value of true

Merge pull request #20 from claudiofahey/issue-19-shutdown-race

7262079

Increase sleep time to avoid a race condition when a test completes

Issue 23: Add option to disable deleting of streams (#25)

f5dc4e2

Issue 24: Add keycloak dependency (#27)

4c41565

* Add option to create scope * Add keycloak dependency

Issue 17: Re-use ByteBuffer for build write payloads with timestamp (#29

2dd9634

) Avoid allocating a new byte buffer for each write for efficiency reasons. Signed-off-by: Raúl Gracia <[email protected]>

RaulGracia requested review from jingerbread, claudiofahey, Tristan1900 and leapsky June 22, 2020 14:22

RaulGracia and others added 5 commits June 25, 2020 16:04

Issue 30: Enable tiering in Pulsar (#31)

4f70de9

Add configuration to enable Tiering in Pulsar. Signed-off-by: Raúl Gracia <[email protected]> Co-authored-by: Malygina <[email protected]>

Keeping final changes for master.

566d355

Signed-off-by: Raúl Gracia <[email protected]>

Set all instances for Kafka, Pulsar and Pravega with the same instanc…

b9092be

…e types. Signed-off-by: Raúl Gracia <[email protected]>

Remove synchronized block from transactional writer.

9fa1e01

Signed-off-by: Raúl Gracia <[email protected]>

Remove synchronized block from transactional writer.

2b35ce7

Signed-off-by: Raúl Gracia <[email protected]>

RaulGracia force-pushed the issue-32-cleanup-dev branch from 9768588 to 2b35ce7 Compare June 25, 2020 14:25

RaulGracia marked this pull request as ready for review June 25, 2020 14:27

claudiofahey requested changes Jun 29, 2020

View reviewed changes

RaulGracia and others added 3 commits July 3, 2020 12:58

Set right version Netty version for Pravega.

df4dd05

Signed-off-by: Raúl Gracia <[email protected]>

[updated] claudiofahey/pravega-tester:0.7.0 from 0.5.1 version

68986db

[changed] pravegaSrcRemote: yes pravegaSrc: https://github.com/praveg…

d79b46f

…a/pravega/releases/download/v{{ pravegaVersion}}/pravega-{{ pravegaVersion }}.tgz

jingerbread closed this Jul 15, 2020

RaulGracia reopened this Jul 15, 2020

jingerbread approved these changes Jul 17, 2020

View reviewed changes

RaulGracia requested review from claudiofahey and removed request for Tristan1900 and leapsky July 17, 2020 14:49

RaulGracia merged commit 4a92e6d into master Jul 17, 2020

RaulGracia mentioned this pull request Jul 17, 2020

Cleanup dev branch to merge to master #32

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 32: Update master branch with contents of dev branch #33

Issue 32: Update master branch with contents of dev branch #33

RaulGracia commented Jun 22, 2020

claudiofahey Jun 29, 2020

RaulGracia Jul 1, 2020

claudiofahey Jul 1, 2020

RaulGracia Jul 1, 2020

jingerbread commented Jul 8, 2020 •

edited

Loading

RaulGracia commented Jul 13, 2020

claudiofahey commented Jul 13, 2020

jingerbread commented Jul 15, 2020 •

edited

Loading

RaulGracia commented Jul 15, 2020

jingerbread commented Jul 15, 2020

RaulGracia commented Jul 15, 2020

jingerbread commented Jul 15, 2020 •

edited

Loading

jingerbread commented Jul 15, 2020

jingerbread commented Jul 15, 2020 •

edited

Loading

jingerbread commented Jul 15, 2020

jingerbread commented Jul 15, 2020

RaulGracia commented Jul 16, 2020

jingerbread commented Jul 16, 2020 •

edited

Loading

jingerbread commented Jul 16, 2020 •

edited

Loading

jingerbread commented Jul 16, 2020 •

edited

Loading

RaulGracia commented Jul 17, 2020

jingerbread commented Jul 17, 2020 •

edited

Loading

Issue 32: Update master branch with contents of dev branch #33

Issue 32: Update master branch with contents of dev branch #33

Conversation

RaulGracia commented Jun 22, 2020

claudiofahey Jun 29, 2020

Choose a reason for hiding this comment

RaulGracia Jul 1, 2020

Choose a reason for hiding this comment

claudiofahey Jul 1, 2020

Choose a reason for hiding this comment

RaulGracia Jul 1, 2020

Choose a reason for hiding this comment

jingerbread commented Jul 8, 2020 • edited Loading

RaulGracia commented Jul 13, 2020

claudiofahey commented Jul 13, 2020

jingerbread commented Jul 15, 2020 • edited Loading

RaulGracia commented Jul 15, 2020

jingerbread commented Jul 15, 2020

RaulGracia commented Jul 15, 2020

jingerbread commented Jul 15, 2020 • edited Loading

jingerbread commented Jul 15, 2020

jingerbread commented Jul 15, 2020 • edited Loading

jingerbread commented Jul 15, 2020

jingerbread commented Jul 15, 2020

RaulGracia commented Jul 16, 2020

jingerbread commented Jul 16, 2020 • edited Loading

jingerbread commented Jul 16, 2020 • edited Loading

jingerbread commented Jul 16, 2020 • edited Loading

RaulGracia commented Jul 17, 2020

jingerbread commented Jul 17, 2020 • edited Loading

jingerbread commented Jul 8, 2020 •

edited

Loading

jingerbread commented Jul 15, 2020 •

edited

Loading

jingerbread commented Jul 15, 2020 •

edited

Loading

jingerbread commented Jul 15, 2020 •

edited

Loading

jingerbread commented Jul 16, 2020 •

edited

Loading

jingerbread commented Jul 16, 2020 •

edited

Loading

jingerbread commented Jul 16, 2020 •

edited

Loading

jingerbread commented Jul 17, 2020 •

edited

Loading