DuckDB 1.1 on windows cannot load extension ? #13848

jgranduel · 2024-09-10T13:27:23Z

What happens?

Installing an extension makes DuckDB REPL stop.
Some extensions being loaded from .duckdbrc, nothing happened at first trial.

> duckdb.exe
v1.1.0 fa5c2fe15f
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D install spatial;
(duckdb terminates)

Is it related to changelog:

"Disable Windows extensions CI until Github actions runners are fixed" by @Mytherin #12479

(I see it just before submitting...)

To Reproduce

Tested with

install spatialite;
install httpfs;
install shellfs;
install prql from community;

with same results.

OS:

Windows-10

DuckDB Version:

1.1.0

DuckDB Client:

duckdb CLI/REPL

Hardware:

No response

Full Name:

jgran

Affiliation:

NA

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

Yes, I have

The text was updated successfully, but these errors were encountered:

szarnyasg · 2024-09-10T13:38:00Z

Hi @jgranduel thanks for opening this issue. I tried to reproduce it on a laptop running Windows 11. I installed DuckDB v1.1.0 via winget. I could install and load the spatial, json extension as well as community extensions. I'm leaving this open – is others are also experiencing this, please comment on this issue and provide details on your configuration (architecture, windows version, how did you install DuckDB, etc.).

carlopi · 2024-09-10T14:03:50Z

I believe the problem is potentially elsewhere, given the first statement on the CLI also executes whatever you have in your ~/.duckdbrc file and/or load the attached database / config.

Could you possibly try something like: SELECT 42; and see if that reproduces? That that would confirm it's about the duckdbrc file.

In any case it would be relevant to list which extensions have been installed at ~/.duckdb/extensions/v1.1.0/*/*, and checking if there are any persisten secrets in the ~/.duckdb/stored_secrets/ folder, and list those (or the extensions they belong to.

Thanks for the informations

Hunknown · 2024-09-10T14:13:33Z

Hi,

I'm using Windows 11 64bits and Python (3.11.5, miniconda), and I also have this issue. I've installed duckdb via pip. I have tried on a fresh new environment.

I've tried spatial and sqlite_scanner.

When I run load spatial, Python crashes right away without any error description.

>>> import duckdb
>>> con = duckdb.connect()
>>> con.sql("load 'assets/extensions/spatial.duckdb_extension'")
(crashes)

I can load sqlite_scanner and add a connection to a database with attach 'xx.db' (type SQLITE), but when I try to run any query on the attached database, Python crashes right away as well.

>>> import duckdb
>>> con = duckdb.connect()
>>> con.sql("load 'assets/extensions/sqlite_scanner.duckdb_extension'")
>>> import sqlite3
>>> sqlite3.connect('test.db')
>>> con.sql("attach 'test.db' as test (type sqlite)")
>>> con.sql("create table test.test (id int)")
(crashes)

I can create and query tables without any issue in pure duckdb (without extensions).

Please note that I downloaded the extensions manually from http://extensions.duckdb.org because of corporate proxy.

Here's the links I've used:

I've checked the folder mentionned by @carlopi but I don't have any stored_secrets folder. My extensions folder was empty, I tried to put the manually downloaded extensions in there, and running the command load spatial (without explicit path) but it still crashes.

carlopi · 2024-09-10T14:24:44Z

Yet another question, can you check what's the answer for:

>>> import platform
>>> platform.system()
>>> platform.processor()
>>> platform.platform()
>>> platform.machine()
>>> platform.version()
>>> platform.uname()

in the Python3 CLI?
Or any other way to detect actual platform? Thanks

Hunknown · 2024-09-10T14:27:05Z

Sure thing, here's the results:

>>> import platform
>>> platform.system()
'Windows'
>>> platform.processor()
'AMD64 Family 25 Model 68 Stepping 1, AuthenticAMD'
>>> platform.platform()
'Windows-10-10.0.22621-SP0'
>>> platform.machine()
'AMD64'
>>> platform.version()
'10.0.22621'
>>> platform.uname()
uname_result(system='Windows', node='PF48-00Y', release='10', version='10.0.22621', machine='AMD64')

Not sure why it says Windows 10, because I'm on Windows 11...

carlopi · 2024-09-10T14:35:43Z

Thanks, yet another question, can you check what's the hash of the files at assets/extensions/sqlite_scanner.duckdb_extension / assets/extensions/spatial.duckdb_extension ?

With compute_hash.py as:

import hashlib
import sys

file=sys.argv[1]
print(file)

with open(file, "rb") as f:
    digest = hashlib.file_digest(f, "sha256")

print(digest.hexdigest())

I get then:

% python3 compute_hash.py path/to/spatial.duckdb_extension
path/to/spatial.duckdb_extension
d1dbdd0f5ceecc5ca0bcb1483c077381d2cb49f27293168ba8ec5597ca69638b

(hash depends on platform, but I can check against the one I know are live)

carlopi · 2024-09-10T14:36:45Z

Note that there might also be two separate issues here at play, so info is relevant also from OP @jgranduel, thanks a lot.

Hunknown · 2024-09-10T14:43:51Z

Thanks for the reply, @carlopi

Here's the hashes for the extensions I've downloaded for 1.1:

.\spatial.duckdb_extension
13ac48bf54ae73eb9780a658bc84b86d81819abf1f0d5674a4e2c45901e825eb

.\sqlite_scanner.duckdb_extension
1657ec59d4d069fc0402dd75d8ea3c229520520cf14d76c8d5dd38b14166cb54

carlopi · 2024-09-10T14:53:01Z

Extensions have the same hashes I would expect, so that's one less variable. I will need to check some other ideas, or someone else on the team might have some questions.

jgranduel · 2024-09-10T18:52:27Z

Just a complement for being clear(er).

I downloaded the new duckDB from release page (this one: https://github.com/duckdb/duckdb/releases/download/v1.1.0/duckdb_cli-windows-amd64.zip).
I installed it beside duckdb-1.0.0.
As nothing happened when I launch duckdb-1.1.0 CLI, I removed .duckdbrc, duckdb worked well.
I reintroduced .duckdbrc. I only have 4 lines in it, including a new prompt with a 🦆 ;-) . It worked well until I added the load extension step. I thought I had to reinstall them one by one as described earlier.
The issue is not the .duckdbrc. It just happened I was loading extensions in it.

I don't use spatialite, but it's the one used in documentation. I tried other ones I know of, and that I was sure they work with my duckdb-1.0.0 install.

carlopi · 2024-09-10T19:13:06Z

Just a semi-random try, @jgranduel: what's the status of the ~/.duckdb/extensions/v1.1.0/windows_amd64/ folder AFTER the failures?

And if not there, what happens if you manually create the folder ~/.duckdb/extensions/v1.1.0/windows_amd64/ and then try again?

Thanks for following up on this.

jgranduel · 2024-09-10T20:16:09Z

Indeed I had no folders ~/.duckdb/extensions/v1.1.0/windows_amd64.
I created them with mkdir and it didn't change anything, sorry!

I won't be able to test in the coming days, or very sparingly, sorry...

AntoineGiraud · 2024-09-11T07:21:58Z

Hi :)

Same issue here on 3 on 5 windows computers !

Also, on the 3 faulty PC only

on 🦆 v1.1.0 : select 1; crash install spatial; crash
on 🦆 v1.0.0 : select 1; ok install spatial; ok
install h3 from community; crash
Crash when loading on Windows Server isaacbrodsky/h3-duckdb#122

👻 buggy ones

Windows 10 Pro - 10, 10.0.19045, AMD64, Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
Windows 10 Pro - 10, 10.0.19045, AMD64, Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
Windows, 11 Pro 23H2 - 10, 10.0.22631, AMD64, Intel64 Family 6 Model 142 Stepping 10, GenuineIntel

🦆 fine ones

Surface Pro 6 Windows 11 Famille 22H2 - 10, 10.0.22621, AMD64, Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
Windows 10 Enterprise 22H2 - 10, 10.0.19045, AMD64, Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
Windows 11 Enterprise 23H2 - 10, 10.0.22631, AMD64, Intel64 Family 6 Model 140 Stepping 1, GenuineIntel

thanks in advence for your investigation 🙏
hope it helps !

carlopi · 2024-09-11T08:08:20Z

Thanks everyone, it's helpful to collect info, @AntoineGiraud in particular, can you possibly run the same experiment with having this environment variable set when you launch duckdb v1.1.0?

SLURM_MEM_PER_NODE=256 ./duckdb

I need to check with the others on what's the most proper way to track this down, thanks for the patience and availability.

AntoineGiraud · 2024-09-11T08:14:43Z

Hi again :)

Still crashes, i tried in my cmd

set SLURM_MEM_PER_NODE=256
duckdb > select 1;

szarnyasg · 2024-09-11T08:58:15Z

Hi @AntoineGiraud, thanks for the information. We'll investigate and get back to you.

lukaseder · 2024-09-11T14:15:23Z

Not sure if this is the same issue, but when using 1.1.0 in Dbeaver (via the duckdb_jdbc driver), then the JVM crashes (including Dbeaver) with these statements:

INSTALL spatial; -- this works
LOAD spatial;    -- this crashes the JVM

The crash report:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffcdf622f58, pid=3132, tid=8700
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.2+13 (21.0.2+13) (build 21.0.2+13-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.2+13 (21.0.2+13-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64)
# Problematic frame:
# C  [msvcp140.dll+0x12f58]
#
# No core dump will be written. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\Users\lukas\jOOQ\3.20\workspace\jOOQ-pro\jOOQ-test-integration\hs_err_pid3132.log
[11.864s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

Let me know if you think this is a separate issue, or if you'd like to see the contents of hs_err_pid3132.log

szarnyasg · 2024-09-11T14:19:47Z

Update: the crash seems to occur when using an old version of the Microsoft Visual C++ Redistributable package.

I upgraded to the latest one from https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170 and the problem went away.

lukaseder · 2024-09-11T14:26:31Z

I upgraded to the latest one from https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170 and the problem went away.

I can confirm, this fixes it for me as well.

Mytherin · 2024-09-11T14:28:32Z

The issue is most likely caused by Github updating their Windows runners, which we use for building our binaries. We'll try to downgrade the runners and see if we can produce binaries that don't require this newer version.

AntoineGiraud · 2024-09-11T15:40:03Z

All good 🦆🚀🥳

Problem solved with msvc upgrade ✅
ps: i had to drop ~\.duckdb\extensions\v1.1.0 folder

It also solved my h3 issue 🗺️
isaacbrodsky/h3-duckdb#122

lukaseder · 2024-09-11T15:45:34Z

Unfortunately, the problem re-appeared for me after a while. Will wait for the fix

szarnyasg · 2024-09-11T18:21:28Z

There are at least two different issues discussed in this thread:

DuckDB crashes for any query, e.g. SELECT 1;. This is fixed by upgrading to the latest MSVC.
Extension loading issues. According to DuckDB 1.1 on windows cannot load extension ? #13848 (comment), it's useful to clean ~\.duckdb\extensions\v1.1.0 and there may be other causes.

RaczeQ · 2024-09-11T18:42:13Z

Same issue on Github Action Windows runner: https://github.com/kraina-ai/quackosm/actions/runs/10810198152/job/29986937048?pr=153#step:9:27
Line 2740 with connection.load_extension("spatial"):
https://github.com/kraina-ai/quackosm/blob/4fc4fb9a0a9f8c40a7f40b731b95d9b993d3a194/quackosm/pbf_file_reader.py#L2740

@szarnyasg

Using windows-latest meant defaulting to a recent Windows SDK, that caused issues such as duckdb#13848 or duckdb#13864 Thanks everyone for reporting this, and @szarnyasg and @Mytherin for helping figuring this out

@szarnyasg

Using windows-latest meant defaulting to a recent Windows SDK, that caused issues such as #13848 or #13864 Issue is, at his core, that we were depending on a Microsoft Visual C++ Redistributable package that is not installed by default on Windows 2019. Thanks everyone for reporting this, and @szarnyasg and @Mytherin for helping figuring this out. This PR provide a way forward for build artifact that will be built, but also allow a blueprint (and the means) on how to rebuild `v1.1.0` release Windows artifacts, that will likely happen tomorrow. This will offer a solution after downloading new artifacts, after we can confirm they work. Other duckdb repositories might have the same problem, and would need to be reviewed.

lukaseder · 2024-09-12T08:10:14Z

2. Extension loading issues. According to DuckDB 1.1 on windows cannot load extension ? #13848 (comment), it's useful to clean ~\.duckdb\extensions\v1.1.0 and there may be other causes.

Ah yes, thanks for pointing this out!

carlopi · 2024-09-16T11:37:24Z

Update:
We built extensions AND artifacts for DuckDB 1.1.0 on Windows using the wrong configuration, and that resulted in building binaries that, once run on Windows versions missing the relevant MSVC package, would lead to segfault.

The problem(s)

This is the generic problem, and it has been manifested in a few different ways:

segfault on any query executed by DuckDB's Windows CLI. Here the problem was with the DuckDB binary itself, broken binaries would segfault on any query, even SELECT 42.
segfault on LOAD of extensions. Here the problem was with the extensions, and would be triggered for example by queries like:
FROM read_parquet('s3://duckdb-blobs/nl-railway/serices-2023.parquet');. Here completing the query requires to load the HTTPFS extension, that is in itself also a binary, than then once executing extension-code would lead to segfault. This might affect both core extensions (such as httpfs) and community extensions (such as h3).

Problems are slightly different, and have slightly different solutions.
Problem 1 regards the binaries or libraries embedding the duckdb code. This is known to affect DuckDB CLI on Windows, DuckDB-ODBC for Windows, and possibly also other clients, but NOT duckdb-python.

Problem 2 regards the extensions that ship optional DuckDB logic, those are in common for duckdb-clients, and hosted at http(s)://extensions.duckdb.org or http(s)://community-extensions.duckdb.org.

This class of problems have been signaled only on DuckDB 1.1.0 when used on a Windows machine. Downgrading to DuckDB v1.0.0 and waiting for the v1.1.1 patch release is also a viable option.

Solutions and Workarounds

Problem 1, if encountered, can be solved in two ways:

bumping the version of the MSVC package, see this comment from Gabor,
re-downloading the relevant binaries. The installation page at https://duckdb.org/docs/installation/ has been updated to hold the relevant links, so select your client, then your platform (= Windows), then other options, then download the relevant binary either via the browser (click and then you will find the link on the Downloads folder) OR via other tools.
Those binaries have been producing by compiling DuckDB v1.1.0 in a Windows 2019 environment instead of a Windows 2022 environment.

Problem 2, if encountered, can be solved by issuing a UPDATE EXTENSIONS on DuckDB startup, without having any extension already loaded. Or by performing FORCE INSTALL extension_name, for example FORCE INSTALL httpfs. Or, and it will be equivalent, by removing the folder at ~/.duckdb/extensions/v1.1.0 (than re-installing the extension depending on needs).

Connected issues

A list of problems that are known to be connected to this one, and falls in one of the two categories, are:

What's next

If you are an user on DuckDB v1.1.0, follow one of the step above. Those might not be needed at all if you are not experiencing any problem.
If you are developing or maintaining a tool based DuckDB, please do check the build environment, you might also be distributing binaries that might then lead to uncontrolled termination of the host process, that is NOT pleasant experience for your users.
If you are developing or maintaining a DuckDB extension, please update to use the latest commit of the v1.1.0 branch of extension-ci-tools, that includes duckdb/extension-ci-tools@01dccd2.

What next

v1.1.1 is planned for next week, we landed already some PR we aim at producing and testing binaries with Windows 2019 and equivalent.

Thanks everyone for signalling those issues, and helping track this problem down.

carlopi · 2024-09-16T11:40:38Z

Closing this for now as there are workarounds and it's solved in nightly Windows CLI builds.

If you are aware of other relevant information, please do comment or reach out. Thanks

DrewScatterday · 2024-09-20T17:03:04Z

I'm still running into Windows issues with duckDB on node.js after bumping MSVC to latest version and deleting .duckdb/extensions

more details here at duckdb/duckdb-spatial#405

jgranduel added the needs triage label Sep 10, 2024

szarnyasg added the under review label Sep 10, 2024

duckdblabs-bot removed the needs triage label Sep 10, 2024

szarnyasg added the reproduced label Sep 11, 2024

duckdblabs-bot removed the under review label Sep 11, 2024

lukaseder mentioned this issue Sep 11, 2024

Upgrade DuckDB dependency to 1.1.1 jOOQ/jOOQ#17235

Closed

AntoineGiraud mentioned this issue Sep 11, 2024

Crash when loading on Windows Server isaacbrodsky/h3-duckdb#122

Closed

RaczeQ mentioned this issue Sep 11, 2024

feat: change minimal duckdb version kraina-ai/quackosm#153

Merged

szarnyasg mentioned this issue Sep 11, 2024

duckdb1.1 crash with open file in windows 10 #13864

Closed

2 tasks

carlopi mentioned this issue Sep 11, 2024

Explicit windows-2019 instead of windows-latest #13883

Merged

This was referenced Sep 12, 2024

ODBC driver version (1.0.0) had an issue on windows server (2019 at least) whereas v0.10.3 works fine duckdb/duckdb-odbc#24

Open

1.1.0，WIndows only, Attach postgres/mysql extention fail. JVM crash. duckdb/duckdb-java#81

Closed

carlopi mentioned this issue Sep 16, 2024

Using read_parquet crash python process when script previously imported pandas #13910

Closed

2 tasks

carlopi closed this as completed Sep 16, 2024

This was referenced Sep 18, 2024

ST_Read crashes duckdb on Windows (but not Linux!) when reading from GeoJSON API duckdb/duckdb-spatial#289

Closed

json extension - fatal error during loading #11395

Open

Maxxen mentioned this issue Sep 19, 2024

DuckDB Crashes when trying to use r-tree spatial index duckdb/duckdb-spatial#405

Closed

Mytherin mentioned this issue Sep 21, 2024

When installing the postgres_scanner plugin, duck db crashes and exits duckdb/duckdb-postgres#267

Closed

2 tasks

DrewScatterday mentioned this issue Sep 23, 2024

Windows spatial extension fails to load duckdb/duckdb-spatial#409

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DuckDB 1.1 on windows cannot load extension ? #13848

DuckDB 1.1 on windows cannot load extension ? #13848

jgranduel commented Sep 10, 2024

szarnyasg commented Sep 10, 2024

carlopi commented Sep 10, 2024 •

edited

Loading

Hunknown commented Sep 10, 2024

carlopi commented Sep 10, 2024

Hunknown commented Sep 10, 2024 •

edited

Loading

carlopi commented Sep 10, 2024

carlopi commented Sep 10, 2024

Hunknown commented Sep 10, 2024

carlopi commented Sep 10, 2024

jgranduel commented Sep 10, 2024

carlopi commented Sep 10, 2024

jgranduel commented Sep 10, 2024

AntoineGiraud commented Sep 11, 2024 •

edited

Loading

carlopi commented Sep 11, 2024

AntoineGiraud commented Sep 11, 2024

szarnyasg commented Sep 11, 2024

lukaseder commented Sep 11, 2024 •

edited

Loading

szarnyasg commented Sep 11, 2024

lukaseder commented Sep 11, 2024

Mytherin commented Sep 11, 2024

AntoineGiraud commented Sep 11, 2024 •

edited

Loading

lukaseder commented Sep 11, 2024

szarnyasg commented Sep 11, 2024

RaczeQ commented Sep 11, 2024

lukaseder commented Sep 12, 2024

carlopi commented Sep 16, 2024 •

edited

Loading

carlopi commented Sep 16, 2024

DrewScatterday commented Sep 20, 2024

DuckDB 1.1 on windows cannot load extension ? #13848

DuckDB 1.1 on windows cannot load extension ? #13848

Comments

jgranduel commented Sep 10, 2024

What happens?

To Reproduce

OS:

DuckDB Version:

DuckDB Client:

Hardware:

Full Name:

Affiliation:

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

Did you include all relevant data sets for reproducing the issue?

Did you include all code required to reproduce the issue?

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

szarnyasg commented Sep 10, 2024

carlopi commented Sep 10, 2024 • edited Loading

Hunknown commented Sep 10, 2024

carlopi commented Sep 10, 2024

Hunknown commented Sep 10, 2024 • edited Loading

carlopi commented Sep 10, 2024

carlopi commented Sep 10, 2024

Hunknown commented Sep 10, 2024

carlopi commented Sep 10, 2024

jgranduel commented Sep 10, 2024

carlopi commented Sep 10, 2024

jgranduel commented Sep 10, 2024

AntoineGiraud commented Sep 11, 2024 • edited Loading

carlopi commented Sep 11, 2024

AntoineGiraud commented Sep 11, 2024

szarnyasg commented Sep 11, 2024

lukaseder commented Sep 11, 2024 • edited Loading

szarnyasg commented Sep 11, 2024

lukaseder commented Sep 11, 2024

Mytherin commented Sep 11, 2024

AntoineGiraud commented Sep 11, 2024 • edited Loading

lukaseder commented Sep 11, 2024

szarnyasg commented Sep 11, 2024

RaczeQ commented Sep 11, 2024

lukaseder commented Sep 12, 2024

carlopi commented Sep 16, 2024 • edited Loading

The problem(s)

Solutions and Workarounds

Connected issues

What's next

What next

carlopi commented Sep 16, 2024

DrewScatterday commented Sep 20, 2024

carlopi commented Sep 10, 2024 •

edited

Loading

Hunknown commented Sep 10, 2024 •

edited

Loading

AntoineGiraud commented Sep 11, 2024 •

edited

Loading

lukaseder commented Sep 11, 2024 •

edited

Loading

AntoineGiraud commented Sep 11, 2024 •

edited

Loading

carlopi commented Sep 16, 2024 •

edited

Loading