Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

特征数据插入问题 #298

Open
liuruifeng-git opened this issue May 26, 2022 · 2 comments
Open

特征数据插入问题 #298

liuruifeng-git opened this issue May 26, 2022 · 2 comments
Assignees

Comments

@liuruifeng-git
Copy link

milvus version 2.0.2
java sdk version 2.0.4

代码:
//插入milvus
List<InsertParam.Field> fields = new ArrayList<>();
fields.add(new InsertParam.Field("features", DataType.FloatVector, featureLists));
fields.add(new InsertParam.Field("date", DataType.Int64, dateLongList));
fields.add(new InsertParam.Field("id", DataType.Int64, dateLongList));
InsertParam insertParam = InsertParam.newBuilder()
.withCollectionName("BidSpaceTest")
.withFields(fields)
.build();
milvusServiceClient.insert(insertParam);

插入了一条数据

数据:[[0.11091897, 0.10981425, 0.10946529, 0.1089742, 0.10914157, 0.108064614, 0.10804585, 0.10828179, 0.10734034, 0.10702038, 0.10663149, 0.10704086, 0.10701183, 0.10654758, 0.10685421, 0.10669913, 0.1064992, 0.10627602, 0.10661164, 0.10728494, 0.10685554, 0.105962515, 0.10686829, 0.10801221, 0.1069447, 0.106962144, 0.106281385, 0.10432063, 0.10262434, 0.10014409, 0.096441545, 0.095015466, 0.09188137, 0.08887424, 0.08656455, 0.08511757, 0.08295897, 0.07980334, 0.07760976, 0.07634725, 0.07645264, 0.075664766, 0.074994124, 0.07825248, 0.07656363, 0.07601002, 0.07278369, 0.070964634, 0.07016153, 0.06848421, 0.06809411, 0.06753272, 0.066838585, 0.06698062, 0.06736541, 0.068205856, 0.069764845, 0.07153347, 0.0736938, 0.07581258, 0.074329704, 0.07707215, 0.080937155, 0.08457597, 0.09076262, 0.098599695, 0.10443987, 0.10933307, 0.11352261, 0.116187304, 0.11698627, 0.120516226, 0.1222908, 0.12383664, 0.12682448, 0.12900138, 0.13056545, 0.13194276, 0.1324476, 0.13245995, 0.13170451, 0.12993094, 0.12835029, 0.12715496, 0.1254788, 0.12357853, 0.121472396, 0.12026113, 0.117506854, 0.11533012, 0.112836026, 0.110219866, 0.10879817, 0.10778891, 0.107614845, 0.1063989]]

attu查看库中的值:
[0.10946529358625412,0.10897420346736908,0.10914157330989838,0.10806461423635483,0.10804585367441177,0.10828179121017456,0.10734034329652786,0.10702037811279297,0.10663148760795593,0.10704085975885391,0.1070118322968483,0.10654757916927338,0.10685420781373978,0.1066991314291954,0.10649920254945755,0.1062760204076767,0.10661163926124573,0.10728494077920914,0.10685554146766663,0.10596251487731934,0.10686828941106796,0.10801220685243607,0.10694470256567001,0.10696214437484741,0.10628138482570648,0.10432063043117523,0.10262434184551239,0.10014408826828003,0.09644154459238052,0.09501546621322632,0.09188137203454971,0.08887424319982529,0.0865645483136177,0.08511757105588913,0.08295896649360657,0.07980334013700485,0.07760976254940033,0.0763472467660904,0.07645264267921448,0.07566476613283157,0.07499412447214127,0.07825247943401337,0.07656362652778625,0.07601001858711243,0.0727836936712265,0.07096463441848755,0.07016152888536453,0.06848420947790146,0.06809411197900772,0.06753271818161011,0.0668385848402977,0.06698062270879745,0.06736540794372559,0.06820585578680038,0.06976484507322311,0.07153347134590149,0.07369379699230194,0.07581257820129395,0.07432970404624939,0.0770721510052681,0.08093715459108353,0.08457597345113754,0.09076262265443802,0.09859969466924667,0.1044398695230484,0.10933306813240051,0.11352261155843735,0.11618730425834656,0.11698626726865768,0.12051622569561005,0.12229079753160477,0.12383663654327393,0.1268244832754135,0.12900137901306152,0.13056544959545135,0.1319427639245987,0.13244760036468506,0.13245995342731476,0.13170450925827026,0.12993094325065613,0.12835028767585754,0.12715496122837067,0.1254788041114807,0.12357853353023529,0.12147239595651627,0.12026113271713257,0.11750685423612595,0.11533012241125107,0.11283602565526962,0.11021986603736877,0.10879816859960556,0.10778891295194626,0.10761484503746033,0.10639890283346176,3.3160085836446607e-38,0]

用attu csv导入后的值:
[0.11091896891593933,0.10981424897909164,0.10946529358625412,0.10897420346736908,0.10914157330989838,0.10806461423635483,0.10804585367441177,0.10828179121017456,0.10734034329652786,0.10702037811279297,0.10663148760795593,0.10704085975885391,0.1070118322968483,0.10654757916927338,0.10685420781373978,0.1066991314291954,0.10649920254945755,0.1062760204076767,0.10661163926124573,0.10728494077920914,0.10685554146766663,0.10596251487731934,0.10686828941106796,0.10801220685243607,0.10694470256567001,0.10696214437484741,0.10628138482570648,0.10432063043117523,0.10262434184551239,0.10014408826828003,0.09644154459238052,0.09501546621322632,0.09188137203454971,0.08887424319982529,0.0865645483136177,0.08511757105588913,0.08295896649360657,0.07980334013700485,0.07760976254940033,0.0763472467660904,0.07645264267921448,0.07566476613283157,0.07499412447214127,0.07825247943401337,0.07656362652778625,0.07601001858711243,0.0727836936712265,0.07096463441848755,0.07016152888536453,0.06848420947790146,0.06809411197900772,0.06753271818161011,0.0668385848402977,0.06698062270879745,0.06736540794372559,0.06820585578680038,0.06976484507322311,0.07153347134590149,0.07369379699230194,0.07581257820129395,0.07432970404624939,0.0770721510052681,0.08093715459108353,0.08457597345113754,0.09076262265443802,0.09859969466924667,0.1044398695230484,0.10933306813240051,0.11352261155843735,0.11618730425834656,0.11698626726865768,0.12051622569561005,0.12229079753160477,0.12383663654327393,0.1268244832754135,0.12900137901306152,0.13056544959545135,0.1319427639245987,0.13244760036468506,0.13245995342731476,0.13170450925827026,0.12993094325065613,0.12835028767585754,0.12715496122837067,0.1254788041114807,0.12357853353023529,0.12147239595651627,0.12026113271713257,0.11750685423612595,0.11533012241125107,0.11283602565526962,0.11021986603736877,0.10879816859960556,0.10778891295194626,0.10761484503746033,0.10639890283346176]

@yhmo yhmo self-assigned this Jun 1, 2022
@yhmo
Copy link
Contributor

yhmo commented Jun 2, 2022

@liuruifeng-git
I could not reproduce this issue.
My test steps:

  1. start milvus v2.0.2
  2. use java sdk v2.0.4 to create a collection and insert 1 row
    the client code:
void test() {
        String randomCollectionName = "test_1";
        int dimension = 96;
        
        // collection schema
        String field1Name = "id";
        String field2Name = "feature";
        String field3Name = "date";
        List<FieldType> fieldsSchema = new ArrayList<>();
        fieldsSchema.add(FieldType.newBuilder()
                .withPrimaryKey(true)
                .withAutoID(false)
                .withDataType(DataType.Int64)
                .withName(field1Name)
                .build());

        fieldsSchema.add(FieldType.newBuilder()
                .withDataType(DataType.FloatVector)
                .withName(field2Name)
                .withDimension(dimension)
                .build());

        fieldsSchema.add(FieldType.newBuilder()
                .withDataType(DataType.Int64)
                .withName(field3Name)
                .build());

        // create collection
        CreateCollectionParam createParam = CreateCollectionParam.newBuilder()
                .withCollectionName(randomCollectionName)
                .withFieldTypes(fieldsSchema)
                .build();

        R<RpcStatus> createR = client.createCollection(createParam);

        // insert data
        Long[] idArr = {999L};
        List<Long> ids = Arrays.asList(idArr);

        Float[] featureArr = {0.11091897f, 0.10981425f, 0.10946529f, 0.1089742f, 0.10914157f, 0.108064614f, 0.10804585f, 0.10828179f, 0.10734034f, 0.10702038f, 0.10663149f, 0.10704086f, 0.10701183f, 0.10654758f, 0.10685421f, 0.10669913f, 0.1064992f, 0.10627602f, 0.10661164f, 0.10728494f, 0.10685554f, 0.105962515f, 0.10686829f, 0.10801221f, 0.1069447f, 0.106962144f, 0.106281385f, 0.10432063f, 0.10262434f, 0.10014409f, 0.096441545f, 0.095015466f, 0.09188137f, 0.08887424f, 0.08656455f, 0.08511757f, 0.08295897f, 0.07980334f, 0.07760976f, 0.07634725f, 0.07645264f, 0.075664766f, 0.074994124f, 0.07825248f, 0.07656363f, 0.07601002f, 0.07278369f, 0.070964634f, 0.07016153f, 0.06848421f, 0.06809411f, 0.06753272f, 0.066838585f, 0.06698062f, 0.06736541f, 0.068205856f, 0.069764845f, 0.07153347f, 0.0736938f, 0.07581258f, 0.074329704f, 0.07707215f, 0.080937155f, 0.08457597f, 0.09076262f, 0.098599695f, 0.10443987f, 0.10933307f, 0.11352261f, 0.116187304f, 0.11698627f, 0.120516226f, 0.1222908f, 0.12383664f, 0.12682448f, 0.12900138f, 0.13056545f, 0.13194276f, 0.1324476f, 0.13245995f, 0.13170451f, 0.12993094f, 0.12835029f, 0.12715496f, 0.1254788f, 0.12357853f, 0.121472396f, 0.12026113f, 0.117506854f, 0.11533012f, 0.112836026f, 0.110219866f, 0.10879817f, 0.10778891f, 0.107614845f, 0.1063989f};
        List<List<Float>> features = new ArrayList<>();
        features.add(Arrays.asList(featureArr));

        Long[] dateArr = {20200602L};
        List<Long> dates = Arrays.asList(dateArr);

        List<InsertParam.Field> fieldsInsert = new ArrayList<>();
        fieldsInsert.add(new InsertParam.Field(field1Name, DataType.Int64, ids));
        fieldsInsert.add(new InsertParam.Field(field2Name, DataType.FloatVector, features));
        fieldsInsert.add(new InsertParam.Field(field3Name, DataType.Int64, dates));

        InsertParam insertParam = InsertParam.newBuilder()
                .withCollectionName(randomCollectionName)
                .withFields(fieldsInsert)
                .build();

        R<MutationResult> insertR = client.insert(insertParam);

        // get collection statistics
        R<GetCollectionStatisticsResponse> statR = client.getCollectionStatistics(GetCollectionStatisticsParam
                .newBuilder()
                .withCollectionName(randomCollectionName)
                .withFlush(true)
                .build());
        assertEquals(R.Status.Success.getCode(), statR.getStatus().intValue());

        GetCollStatResponseWrapper stat = new GetCollStatResponseWrapper(statR.getData());
        System.out.println("Collection row count: " + stat.getRowCount());
    }
  1. start Attu service
    docker run -p 8000:3000 -e HOST_URL=http://{ machine IP }:8000 -e MILVUS_URL={ machine IP}:19530 zilliz/attu:latest

  2. load the collection in Attu, query the vector by "id == 999", the result is:
    [0.11091896891593933,0.10981424897909164,0.10946529358625412,0.10897420346736908,0.10914157330989838,0.10806461423635483,0.10804585367441177,0.10828179121017456,0.10734034329652786,0.10702037811279297,0.10663148760795593,0.10704085975885391,0.1070118322968483,0.10654757916927338,0.10685420781373978,0.1066991314291954,0.10649920254945755,0.1062760204076767,0.10661163926124573,0.10728494077920914,0.10685554146766663,0.10596251487731934,0.10686828941106796,0.10801220685243607,0.10694470256567001,0.10696214437484741,0.10628138482570648,0.10432063043117523,0.10262434184551239,0.10014408826828003,0.09644154459238052,0.09501546621322632,0.09188137203454971,0.08887424319982529,0.0865645483136177,0.08511757105588913,0.08295896649360657,0.07980334013700485,0.07760976254940033,0.0763472467660904,0.07645264267921448,0.07566476613283157,0.07499412447214127,0.07825247943401337,0.07656362652778625,0.07601001858711243,0.0727836936712265,0.07096463441848755,0.07016152888536453,0.06848420947790146,0.06809411197900772,0.06753271818161011,0.0668385848402977,0.06698062270879745,0.06736540794372559,0.06820585578680038,0.06976484507322311,0.07153347134590149,0.07369379699230194,0.07581257820129395,0.07432970404624939,0.0770721510052681,0.08093715459108353,0.08457597345113754,0.09076262265443802,0.09859969466924667,0.1044398695230484,0.10933306813240051,0.11352261155843735,0.11618730425834656,0.11698626726865768,0.12051622569561005,0.12229079753160477,0.12383663654327393,0.1268244832754135,0.12900137901306152,0.13056544959545135,0.1319427639245987,0.13244760036468506,0.13245995342731476,0.13170450925827026,0.12993094325065613,0.12835028767585754,0.12715496122837067,0.1254788041114807,0.12357853353023529,0.12147239595651627,0.12026113271713257,0.11750685423612595,0.11533012241125107,0.11283602565526962,0.11021986603736877,0.10879816859960556,0.10778891295194626,0.10761484503746033,0.10639890283346176]

@zhansg
Copy link

zhansg commented Aug 15, 2022

简单说就是attu建的表有bug,插入的标量、向量字段都会被改变。java sdk建的表没问题

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants