Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nvfuser::ComputeAtMap::computeConcreteId throws No potential concrete_id's found #3542

Closed
jjsjann123 opened this issue Dec 7, 2024 · 1 comment

Comments

@jjsjann123
Copy link
Collaborator

hit this assert

" INTERNAL ASSERT FAILED at "/opt/pytorch/nvfuser/csrc/compute_at_map.cpp":1115, please report a bug with repro script to NVFuser at https://github.com/NVIDIA/Fuser/issues. No potential concrete_id's found for disjoint set { iS65{( ceilDiv(( i0 * 4 ), 128) )}; iS81{( ceilDiv(( i2 * i0 ), 128) )}; iS76{( ceilDiv(( i2 * i0 ), 128) )}; iS47{( ceilDiv(( i2 * i0 ), 128) )}; iS89{( ceilDiv(i0, 128) )}; iS85{( ceilDiv(i0, 128) )}; iS53{( ceilDiv(i0, 128) )}; iS41{( ceilDiv(( i2 * i0 ), 128) )}; iS35{( ceilDiv(( i2 * i0 ), 128) )}; iS59{( ceilDiv(( i0 * 4 ), 128) )}; iS71{( ceilDiv(( i0 * 4 ), 128) )} }

Repro test:

TEST_F(PointwiseTest, DomainMapFactory) {
  auto fusion_ptr = std::make_unique<Fusion>();
  auto fusion = fusion_ptr.get();
  FusionGuard fg(fusion);

  TensorView* tv0 = makeContigTensor(1);
  fusion->addInput(tv0);
  TensorView* tv1 = makeContigTensor(2);
  fusion->addInput(tv1);

  auto tv2 = broadcast(tv0, {true, true, false});
  auto tv3 = add(tv2, tv1);
  fusion->addOutput(tv3);

  auto size_val = IrBuilder::create<Val>(4.0, DataType::Int);
  auto one_val = IrBuilder::create<Val>(1, DataType::Int);
  auto tv4 = rand({size_val, one_val, tv0->axis(0)->extent()}, DataType::Float);
  auto tv5 = add(tv2, tv4);
  fusion->addOutput(tv5);

  FusionExecutorCache executor_cache(std::move(fusion_ptr));

  auto options = at::TensorOptions().dtype(at::kFloat).device(at::kCUDA, 0);
  at::Tensor input0 =
            at::empty_strided({25}, {1}, options);
  at::Tensor input1 =
            at::empty_strided({7, 25}, {25, 1}, options);
  auto cg_outputs = executor_cache.runFusionWithInputs({input0, input1});
  // EXPECT_EQ(getVecSizeForPointwise(executor_cache), 4);
  testValidate(fusion, cg_outputs, {input0, input1}, __LINE__, __FILE__);
}

I'm leaning towards this is actually an issue with pointwise scheduler trying to propagate transformation from an illegal reference tv. Opening the issue to track this for now

@jjsjann123
Copy link
Collaborator Author

closing this one, realizing I already opened one in #2514 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant