No joy with floating licenses in production

[fmg-pete]

Hi, late last year I thought I had our keygen floating license working, and in the small amount of testing I did across several machines it seemed to work as expected. If I set the max machines to, say 3, my app would show a "license quota exceeded" modal if I attempted to start a 4th instance of the app. I based my setup / config off this guide:

https://keygen.sh/docs/choosing-a-licensing-model/floating-licenses/

And code here:

https://github.com/keygen-sh/example-csharp-license-activation/blob/master/Program.cs

Last week we rolled out our Windows app across a small team, and that's when the problems began. I don't quite understand what is happening in a setting where some machines have been validated and others are added later. They all seem to randomly stall before sending their first ping, they validate but fail after. I turned on some debugging on the license check, and on the first validation ( when a user would enter a license key ) I get back "FINGERPRINT_SCOPE_MISMATCH" in the status. I've tried all manner of relaxing settings to no avail - Protected "OFF", Strict "OFF", Fingerprint Scope FALSE, and so on, and so on.

Does anybody have any experience with getting this working in a real-world setting? The combination of correct settings seems like a bit of a holy grail at the moment!

Thanks in advance. :)

[ezekg]

The FINGERPRINT_SCOPE_MISMATCH validation code means that the license has at least 1 machine activated, but none match the fingerprint supplied during validation. You should treat this validation code the same as you treat the NO_MACHINES and NO_MACHINE codes, which mean the license has no activated licenses yet, i.e. you should activate the machine.

[fmg-pete]

Hi Zeke, thanks for the rapid reply, I realised what the date was ( 4th July in the 'States, 5th here! ) about 5 minutes after typing it! :O

I was already doing if (NO_MACHINE || NO_MACHINES || FINGERPRINT_SCOPE_MISMATCH) -> Activate, so it's not that. I am checking for a data.id in the Activation response before showing the "Success" view, so maybe there is an async issue there. At this stage though I can't tell if there is some misconfiguration or misunderstanding of the flow that's causing it.

[fmg-pete]

I really don't understand what is happening now. I have a policy restricted to two machines, both have validated licenses and show as active machines in Dashboard. When I make this call from a third machine, with no license key yet entered:

       ```
       var client = new RestClient("https://api.keygen.sh/v1/accounts/" + accountID);
        var request = new RestRequest("licenses/actions/validate-key").AddJsonBody(
        new
        {
            meta = new
            {
                scope = new { fingerprint = fp },
                key = licenseKey
            }
        });

        var p = await client.ExecutePostAsync(request, default);
       ```

The meta.code response is "FINGERPRINT_SCOPE_MISMATCH". I was expecting "TOO_MANY_MACHINES". Do I have to to ALWAYS activate it on a "FINGERPRINT_SCOPE_MISMATCH", and validate it again? And then deactivate it again? Maybe that's what I'm missing... this flow seems to suggest it:

https://keygen.sh/docs/choosing-a-licensing-model/floating-licenses/

EDIT: With a valid key is just sails through validation exceeding the MAX MACHINES value without issue:

meta.valid: True
code:"VALID"

It only seems to work correctly for invalid keys, which do return as NOT VALID.

[ezekg]

Something doesn't seem right with your integration (or possibly your policy config) if you're able to exceed the max machine limit. Are you able to share a request ID or two with me showing this happening? That'd make debugging this a bit easier on my end. You can find your request logs here.

The only policy config that would effect this would be an overage strategy — so make sure that's set to NO_OVERAGE if you don't want to allow overages.

[fmg-pete]

NO_OVERAGE is True, the rest of the Policy is based on the Floating License documentation, and it doesn't deviate much from that. I made some headway sorting some of the validate / activate flow issues related to machine limits, and I think I might have solved the exceed machine limit issue. But now I am stuck on the scenario where one machine has activated, and there are enough floating licenses available, but it fails. Initially I got TWO errors returned from the POST to the machines endpoint: "MACHINE_LIMIT_EXCEEDED", and "FINGERPRINT_TAKEN". After a couple of tries that stopped, and instead I just get "HEARTBEAT_DEAD" for any subsequent requests. It's as if once the machine has validated the licensing fails to recognise the fingerprint, and it never lets it use it again. I have no idea how that is meant to work. I've also ended up writing conditions within conditions to test the scenarios I have been seeing, which has made for some pretty awful code :(.

I would have expected if the POST request to the validation endpoint (licenses/actions/validate-key) had the fingerprint in it ( I checked logs... it does ) then the back end would know you were just reactivating a machine using that fingerprint. If that isn't the case, how do I handle that?

[ezekg]

It sounds like you're trying to activate a machine that is already activated, which is going to return a conflict error (and is largely superfluous). You should be validating before activating, and only activating if the fingerprint is not currently activated (i.e. the validation codes discussed above). If the machine is already activated, the license validation code should be VALID, and thus there's no need to activate again.

Please share your account ID so that I can take a look at your request logs to determine what's happening here with your activation flow. If you don't want to share it here (it's safe to share but understand you may not want to), feel free to email it to [email protected].

Also, feel free to share code via email if you think that'd be helpful.

[ezekg]

Thanks for the additional info via email. I'll keep responding here for the future reference of others.

It looks like what is happening here is that your machines are dying in Keygen, but in reality, the machine is still active. To me, the issue would signal that your heartbeat process is not making timely heartbeats. In addition, from the logs, it looks like you're also attempting to activate on the HEARTBEAT_DEAD validation code, which will result in an activation error since although the machine is dead, is hasn't been culled yet. So it's still in the license's machine list. Thus, you will receive a FINGERPRINT_TAKEN error during activation until the existing machine has been culled (which can take a few minutes after death). The reactivation would eventually work, as soon as the machine is culled (just as you said is happening).

On HEARTBEAT_DEAD, you should either attempt to resurrect the machine, or you should kill the entire process because it had a problem, failing to send a heartbeat in a timely manner, indicating that there's an application performance or state issue. You should not attempt to reactivate on HEARTBEAT_DEAD, because it's already activated.

[fmg-pete]

You're absolutely right... the "DEAD" machine is still in the list, but it could remain there for any time ( unknown to the user ) based on what's been configured.

I'm sending heartbeats at 5 minute intervals, and AFAICT that's working as expected. Any machine I leave running for hours maintains its place in the machines list, and dutifully drops off 5 minutes after I close the application. The "DEAD" scenario is getting triggered when a machine that had a valid license key, for whatever reason, no longer has one. This might seem like an edge case, but I can think of a number of cases where I license key on an existing machine may either change, or be lost. What I am trying to do there is prompt them to enter the correct key, validate it, and get them back to work.

I think the problem is understanding all of the unforeseen scenarios that happen in the real world, and trying to come up with a configuration and code that mitigates them. The build of our app from last year behaved fine... because I tested against the expected flows and used it "properly" ;) . The team we gave that build to didn't... they did what real users do ( break stuff! ). Trying to figure out the correct combination of Keygen configuration and client-side behaviour is turning out to be waaay more subtle and complicated than it first appears.

I'll try fiddling with the culling / resurrection today.

[ezekg]

Looking at your logs, I don't see that. I see your application attempting to activate right after a HEARTBEAT_DEAD validation code is reached. You should only activate on the following validation codes: NO_MACHINE, NO_MACHINES and FINGERPRINT_SCOPE_MISMATCH. If you know the client's IP, you can actually look at their request logs to reproduce the issue exactly.

If it'd help, please share your code with me via email and I can more easily tell you what's going wrong in your flow.

p.s. if your heartbeat window is 5 minutes, I'd recommend sending heartbeats every 4 minutes, to account for any lag in the network or an out-of-sync clock even. That way your machines don't randomly "die" just because they became slightly out-of-sync time-wise, or their network connectivity suffers for a short period. Give it a buffer of at least 30 seconds.

[fmg-pete]

I figured it out. Deactivating on app exit was the missing link. The dead heartbeat issue became worse when I cranked back the hearbeat duration in the policy I was using for testing ( to 5 minutes ). It could still be a problem at license renewal time, we'll have to deal with it manually by deleting any dead machines. We might have to look into automating that later, for now we'll just allow Keygen to cull them after 30 minutes.

Now working on a dashboard that will allow admins to generate licenses, but that's a whole 'nother story. :)