Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: enable anchoring on recon mode #3185

Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
f8c8bd1
chore: enable anchoring on recon mode
JulissaDantes Mar 15, 2024
3b6a068
chore: make tests support enabled recon
JulissaDantes Mar 17, 2024
e21f329
chore: expose recon in package index
JulissaDantes Mar 18, 2024
8a74337
fix: update tests
JulissaDantes Mar 20, 2024
674d69d
chore: skip tests using tile docs
JulissaDantes Mar 21, 2024
b65ad46
chore: remove console log
JulissaDantes Mar 21, 2024
692c730
chore: restore skip describe
JulissaDantes Mar 21, 2024
e6e3069
chore: remove unused library
JulissaDantes Mar 21, 2024
517ac32
refactor: model instance document test
JulissaDantes Mar 21, 2024
bc68b0a
chore: refactor beforeAll hook
JulissaDantes Mar 21, 2024
aa7e94a
fix: test delay
JulissaDantes Mar 22, 2024
60696ad
Merge branch 'develop' into feature/ws2-3129-enable-anchoring-as-part…
JulissaDantes Mar 25, 2024
b9a4fdb
chore: update tests
JulissaDantes Mar 25, 2024
6ea0c7f
chore: run lint
JulissaDantes Mar 25, 2024
0d28018
chore: update tests
JulissaDantes Mar 25, 2024
c525ff9
chore: update tests
JulissaDantes Mar 25, 2024
31eba9e
chore: remove .only
JulissaDantes Mar 25, 2024
c058d42
feat: add test delay
JulissaDantes Mar 26, 2024
df32401
fix: post merge fix
JulissaDantes Mar 26, 2024
82f3c32
fix: add delay to test
JulissaDantes Mar 26, 2024
e9872ae
fix: update tests
JulissaDantes Mar 26, 2024
f8fa7e9
Merge branch 'develop' into feature/ws2-3129-enable-anchoring-as-part…
JulissaDantes Mar 28, 2024
cdd99b1
fix: add needed delay time
JulissaDantes Mar 28, 2024
4561c7f
chore: run lint
JulissaDantes Mar 28, 2024
c07bc3a
feat: update default creation values
JulissaDantes Mar 29, 2024
cda3492
Merge branch 'develop' into feature/ws2-3129-enable-anchoring-as-part…
JulissaDantes Mar 29, 2024
73108bb
refactor: replace delays for waitFor
JulissaDantes Mar 29, 2024
43cca09
refactor: change test
JulissaDantes Mar 29, 2024
b250250
fix: update test
JulissaDantes Mar 29, 2024
14d58fd
refactor: test hooks
JulissaDantes Mar 29, 2024
5f29187
fix: undo anchor added delay
JulissaDantes Apr 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ test('Dont process the same entry multiple times concurrently', async () => {
async function* overEntries(): AsyncGenerator<number> {
do {
// Need this sleep or else the Node runtime might never switch back to the "process" function.
await CommonTestUtils.delay(1)
await CommonTestUtils.delay(5)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ukstv This is how I solved it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the issue you were seeing? This change really shouldn't be necessary. This sleep isn't actually making the test wait for anything to happen, it's just so that the Node event loop has an interrupt point so other jobs can get scheduled, otherwise the generator can just emit events forever without the task to process those events ever running. But increasing the duration of the sleep should have no affect on the rest of the system or the test, unless I really misunderstood something when I wrote this test.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming this test failure was the issue: https://app.circleci.com/pipelines/github/ceramicnetwork/js-ceramic/13094/workflows/f7e8914c-f718-40e3-bfd6-4ee07174026c/jobs/18766

Put together a PR that I believe should fix the race condition in the test: #3192

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes that was the issue, thanks for the PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this change be reverted now that #3192 is in?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping

for (const e of entries) yield e
} while (true)
}
Expand Down
1 change: 1 addition & 0 deletions packages/core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@ export * from './store/ikv-store.js'
export * from './ceramic.js'
export * from './pubsub/pubsub-message.js'
export * from './utils.js'
export * from './recon.js'

export { ProvidersCache } from './providers-cache.js'
3 changes: 1 addition & 2 deletions packages/core/src/recon.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,6 @@ export class ReconApi extends Observable<ReconEventFeedResponse> implements IRec
if (!this.enabled) {
throw new Error(`Recon: disabled, not registering interest in model ${model.toString()}`)
}

try {
await this.#sendRequest(this.#url + `/ceramic/interests/model/${model.toString()}`, {
method: 'POST',
Expand Down Expand Up @@ -195,7 +194,7 @@ export class ReconApi extends Observable<ReconEventFeedResponse> implements IRec
return of({ events: [], cursor: initialCursor, first: true }).pipe(
// projects the starting event to an Observable that emits the next events. Then it recursively projects each event to an Observable that emits the next event
expand((prev) => {
// creates an observable that emits the next event after a ceratin delay (pollInterval) unless this is the first event
// creates an observable that emits the next event after a certain delay (pollInterval) unless this is the first event
return timer(prev.first ? 0 : this.#pollInterval).pipe(
// concat map is used to ensure that the next event is only emitted after the previous event has been processed
concatMap(() =>
Expand Down
5 changes: 0 additions & 5 deletions packages/core/src/state-management/repository.ts
Original file line number Diff line number Diff line change
Expand Up @@ -647,11 +647,6 @@ export class Repository {
return
}

if (process.env.CERAMIC_RECON_MODE) {
// TODO(WS1-1471): Enable anchoring when in Recon (Prime) mode.
return
}

const carFile = await this.#deps.anchorRequestCarBuilder.build(state$.id, state$.tip)
const anchorEvent = await this.anchorService.requestAnchor(carFile)
// Don't wait on handling the anchor event, let that happen in the background.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,7 @@ describe('ModelInstanceDocument API http-client tests', () => {
expect(doc.state.log.length).toEqual(2)
expect(doc.state.log[0].type).toEqual(EventType.INIT)
expect(doc.state.log[1].type).toEqual(EventType.DATA)
if (!process.env.CERAMIC_RECON_MODE) {
// TODO (WS1-1471): Re-enable this check even in Recon mode
expect(doc.state.anchorStatus).toEqual(AnchorStatus.PENDING)
}
expect(doc.state.anchorStatus).toEqual(AnchorStatus.PENDING)
})

test(`Can create deterministic doc with create method`, async () => {
Expand All @@ -141,10 +138,7 @@ describe('ModelInstanceDocument API http-client tests', () => {
expect(doc.state.log.length).toEqual(2)
expect(doc.state.log[0].type).toEqual(EventType.INIT)
expect(doc.state.log[1].type).toEqual(EventType.DATA)
if (!process.env.CERAMIC_RECON_MODE) {
// TODO (WS1-1471): Re-enable this check even in Recon mode
expect(doc.state.anchorStatus).toEqual(AnchorStatus.PENDING)
}
expect(doc.state.anchorStatus).toEqual(AnchorStatus.PENDING)
})

test(`Creating doc with SINGLE accountRelation non-deterministically should fail `, async () => {
Expand Down Expand Up @@ -260,10 +254,8 @@ describe('ModelInstanceDocument API multi-node tests', () => {

const docState = doc.state
const loadedState = loaded.state
// TODO(WS1-1471): Enable anchoring when in Recon (Prime) mode.
expect(docState.anchorStatus).toEqual(
process.env.CERAMIC_RECON_MODE ? AnchorStatus.NOT_REQUESTED : AnchorStatus.PENDING
)

expect(docState.anchorStatus).toEqual(AnchorStatus.PENDING)
expect(loadedState.anchorStatus).toEqual(AnchorStatus.NOT_REQUESTED)
delete docState.anchorStatus
delete loadedState.anchorStatus
Expand All @@ -281,10 +273,8 @@ describe('ModelInstanceDocument API multi-node tests', () => {

const docState = doc.state
const loadedState = loaded.state
// TODO(WS1-1471): Enable anchoring when in Recon (Prime) mode.
expect(docState.anchorStatus).toEqual(
process.env.CERAMIC_RECON_MODE ? AnchorStatus.NOT_REQUESTED : AnchorStatus.PENDING
)

expect(docState.anchorStatus).toEqual(AnchorStatus.PENDING)
expect(loadedState.anchorStatus).toEqual(AnchorStatus.NOT_REQUESTED)
delete docState.anchorStatus
delete loadedState.anchorStatus
Expand Down
46 changes: 23 additions & 23 deletions packages/stream-tests/src/__tests__/model-instance-document.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { jest, test, expect, describe, beforeAll, afterAll } from '@jest/globals'
import getPort from 'get-port'
import { AnchorStatus, EventType, IpfsApi } from '@ceramicnetwork/common'
import { AnchorStatus, EventType, IpfsApi, Networks } from '@ceramicnetwork/common'
import { Utils as CoreUtils } from '@ceramicnetwork/core'
import { createIPFS, swarmConnect } from '@ceramicnetwork/ipfs-daemon'
import {
Expand Down Expand Up @@ -81,9 +81,6 @@ const MODEL_WITH_RELATION_DEFINITION: ModelDefinition = {
},
}

// TODO(WS1-1471): These tests should be enabled once anchoring works in Recon mode
const testIfV3ShouldPassWithAnchoring = process.env.CERAMIC_RECON_MODE ? test.skip : test

describe('ModelInstanceDocument API http-client tests', () => {
jest.setTimeout(1000 * 30)

Expand Down Expand Up @@ -166,10 +163,8 @@ describe('ModelInstanceDocument API http-client tests', () => {
expect(doc.metadata.unique).toBeInstanceOf(Uint8Array)
expect(doc.state.log.length).toEqual(1)
expect(doc.state.log[0].type).toEqual(EventType.INIT)
if (!process.env.CERAMIC_RECON_MODE) {
// TODO (WS1-1471): Re-enable this check even in Recon mode
expect(doc.state.anchorStatus).toEqual(AnchorStatus.PENDING)
}
expect(doc.state.anchorStatus).toEqual(AnchorStatus.PENDING)

expect(doc.metadata.model.toString()).toEqual(model.id.toString())
await expect(TestUtils.isPinned(ceramic.admin, doc.id)).resolves.toBeTruthy()
await expect(TestUtils.isPinned(ceramic.admin, doc.metadata.model)).resolves.toBeTruthy()
Expand All @@ -187,10 +182,8 @@ describe('ModelInstanceDocument API http-client tests', () => {
expect(docWithRelation.metadata.unique).toBeInstanceOf(Uint8Array)
expect(docWithRelation.state.log.length).toEqual(1)
expect(docWithRelation.state.log[0].type).toEqual(EventType.INIT)
if (!process.env.CERAMIC_RECON_MODE) {
// TODO (WS1-1471): Re-enable this check even in Recon mode
expect(docWithRelation.state.anchorStatus).toEqual(AnchorStatus.PENDING)
}
expect(docWithRelation.state.anchorStatus).toEqual(AnchorStatus.PENDING)

expect(docWithRelation.metadata.model.toString()).toEqual(modelWithRelation.id.toString())
await expect(TestUtils.isPinned(ceramic.admin, docWithRelation.id)).resolves.toBeTruthy()
await expect(
Expand Down Expand Up @@ -272,7 +265,7 @@ describe('ModelInstanceDocument API http-client tests', () => {
expect(docWithRelation.content.optionalLinkedDoc).toBe(docID)
})

testIfV3ShouldPassWithAnchoring('Anchor genesis', async () => {
test('Anchor genesis', async () => {
const doc = await ModelInstanceDocument.create(ceramic, CONTENT0, midMetadata)
expect(doc.state.anchorStatus).toEqual(AnchorStatus.PENDING)

Expand All @@ -286,7 +279,7 @@ describe('ModelInstanceDocument API http-client tests', () => {
expect(doc.content).toEqual(CONTENT0)
})

testIfV3ShouldPassWithAnchoring('Anchor after updating', async () => {
test('Anchor after updating', async () => {
const doc = await ModelInstanceDocument.create(ceramic, CONTENT0, midMetadata)
expect(doc.state.anchorStatus).toEqual(AnchorStatus.PENDING)
await doc.replace(CONTENT1)
Expand All @@ -303,7 +296,7 @@ describe('ModelInstanceDocument API http-client tests', () => {
expect(doc.content).toEqual(CONTENT1)
})

testIfV3ShouldPassWithAnchoring('multiple updates', async () => {
test('multiple updates', async () => {
const doc = await ModelInstanceDocument.create(ceramic, CONTENT0, midMetadata)
await doc.replace(CONTENT1)

Expand Down Expand Up @@ -350,7 +343,7 @@ describe('ModelInstanceDocument API http-client tests', () => {
).rejects.toThrow(/Attempting to create a ModelInstanceDocument with an invalid DID string/)
})

testIfV3ShouldPassWithAnchoring('Can load a stream', async () => {
test('Can load a stream', async () => {
const doc = await ModelInstanceDocument.create(ceramic, CONTENT0, midMetadata)
await doc.replace(CONTENT1)
await CoreUtils.anchorUpdate(core, doc)
Expand Down Expand Up @@ -428,10 +421,7 @@ describe('ModelInstanceDocument API http-client tests', () => {
})
})

// should pass on v4 as soon as recon is integrated and cross-node syncing works.
const describeIfV3ShouldPass = process.env.CERAMIC_RECON_MODE ? describe.skip : describe

describeIfV3ShouldPass('ModelInstanceDocument API multi-node tests', () => {
describe('ModelInstanceDocument API multi-node tests', () => {
jest.setTimeout(1000 * 30)

let ipfs0: IpfsApi
Expand All @@ -442,16 +432,26 @@ describeIfV3ShouldPass('ModelInstanceDocument API multi-node tests', () => {
let midMetadata: ModelInstanceDocumentMetadataArgs

beforeAll(async () => {
ipfs0 = await createIPFS()
ipfs1 = await createIPFS()
ipfs0 = await createIPFS({
rust: {
type: 'binary',
JulissaDantes marked this conversation as resolved.
Show resolved Hide resolved
network: Networks.INMEMORY,
},
})
ipfs1 = await createIPFS({
rust: {
type: 'binary',
network: Networks.INMEMORY,
},
})
await swarmConnect(ipfs0, ipfs1)

ceramic0 = await createCeramic(ipfs0)
ceramic1 = await createCeramic(ipfs1)

model = await Model.create(ceramic0, MODEL_DEFINITION)
midMetadata = { model: model.id }
}, 12000)
}, 80000)

afterAll(async () => {
await ceramic0.close()
Expand Down
Loading