Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stand alone auto increment #6

Open
wants to merge 36 commits into
base: master
Choose a base branch
from
+846 −147
Open

Stand alone auto increment #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
6e6e7a5
Influxdb service adapts to distributed MPP framework
Jul 26, 2022
d32c0dc
fix InfluxFirstFunction,InfluxLastFunction,InfluxMeanFunction
Jul 26, 2022
9e24ad1
Merge branch 'apache:master' into influxdb-service-mpp
KeeProMise Jul 26, 2022
ec08137
Create InfluxDB Protocol Test
KeeProMise Jul 27, 2022
e47bb08
Using the '.*' form of import should be avoided
Jul 27, 2022
57da703
Merge remote-tracking branch 'origin/influxdb-service-mpp' into influ…
Jul 27, 2022
19afe30
spotless apply
Jul 28, 2022
34d4642
influxdb mpp test
Jul 28, 2022
21fb41c
Merge remote-tracking branch 'upstream/master' into influxdb-service-mpp
Jul 28, 2022
3485075
influxdb mpp test
Jul 28, 2022
b688c4d
Modified to rely on an AbstractInfluxDBMetaManager
Jul 28, 2022
9ba89db
Merge branch 'apache:master' into new_influxdb_Mpp
KeeProMise Jul 29, 2022
e0241b6
Generate tag timestamps using AtomicLong
Jul 29, 2022
1ef83e8
Merge branch 'apache:master' into new_influxdb_Mpp
KeeProMise Jul 29, 2022
5b94822
Merge remote-tracking branch 'origin/new_influxdb_Mpp' into influxdb-…
Jul 29, 2022
3482ca2
IDtable add getDiskSchemaEntries()
Aug 16, 2022
2a6b394
Merge remote-tracking branch 'upstream/master' into idtable-disk_read
Aug 16, 2022
9845157
IDtable add deleteTimeserie method
Aug 16, 2022
73b8afb
Merge remote-tracking branch 'upstream/master' into idtable-disk_read
Aug 16, 2022
6cf0506
IDtable add deleteTimeserie method
Aug 18, 2022
fff9512
IDtable add deleteTimeserie method
Aug 18, 2022
790de6e
IDtable add deleteTimeserie method
Aug 18, 2022
d82524f
add AutoIncrementDeviceID
Aug 20, 2022
83b2e04
add AutoIncrementDeviceID use idtable
Aug 22, 2022
db56e67
Merge branch 'apache:master' into master
KeeProMise Aug 23, 2022
f350676
StandAloneAutoIncDeviceID use List
Aug 23, 2022
a5e827a
Merge remote-tracking branch 'origin/master' into dev-idtable
Aug 23, 2022
8210f7f
no
Aug 23, 2022
6d43024
Merge remote-tracking branch 'origin/master'
Aug 23, 2022
5d35f3b
Merge remote-tracking branch 'origin/master' into dev-idtable
Aug 23, 2022
63b6b0b
Stand-alone version supports auto-increment id
Aug 23, 2022
786d430
Merge remote-tracking branch 'origin/master' into standAlone_AutoIncr…
Aug 26, 2022
6abc55f
Modify code comments
Aug 26, 2022
9959aeb
Modify recover
Aug 27, 2022
6900d0a
ADD IDTableAutoIncImpl
Aug 29, 2022
b499ee3
modify
Aug 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion integration/src/test/java/org/apache/iotdb/db/integration/IoTDBQueryWithIDTableIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
package org.apache.iotdb.db.integration;

import org.apache.iotdb.db.conf.IoTDBDescriptor;
import org.apache.iotdb.db.metadata.idtable.entry.DeviceIDFactory;
import org.apache.iotdb.db.metadata.idtable.deviceID.DeviceIDFactory;
import org.apache.iotdb.integration.env.ConfigFactory;
import org.apache.iotdb.integration.env.EnvFactory;
import org.apache.iotdb.itbase.category.LocalStandaloneTest;
Expand Down
2 changes: 1 addition & 1 deletion server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
import org.apache.iotdb.db.exception.metadata.StorageGroupNotSetException;
import org.apache.iotdb.db.exception.query.QueryProcessException;
import org.apache.iotdb.db.exception.runtime.StorageEngineFailureException;
import org.apache.iotdb.db.metadata.idtable.entry.DeviceIDFactory;
import org.apache.iotdb.db.metadata.idtable.deviceID.DeviceIDFactory;
import org.apache.iotdb.db.metadata.mnode.IMeasurementMNode;
import org.apache.iotdb.db.metadata.mnode.IStorageGroupMNode;
import org.apache.iotdb.db.qp.physical.crud.InsertPlan;
Expand Down
2 changes: 1 addition & 1 deletion server/src/main/java/org/apache/iotdb/db/engine/flush/MemTableFlushTask.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
import org.apache.iotdb.db.engine.memtable.IWritableMemChunk;
import org.apache.iotdb.db.engine.memtable.IWritableMemChunkGroup;
import org.apache.iotdb.db.exception.runtime.FlushRunTimeException;
import org.apache.iotdb.db.metadata.idtable.entry.IDeviceID;
import org.apache.iotdb.db.metadata.idtable.deviceID.IDeviceID;
import org.apache.iotdb.db.rescon.SystemInfo;
import org.apache.iotdb.db.service.metrics.MetricService;
import org.apache.iotdb.db.service.metrics.enums.Metric;
Expand Down
4 changes: 2 additions & 2 deletions server/src/main/java/org/apache/iotdb/db/engine/memtable/AbstractMemTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@
import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
import org.apache.iotdb.db.exception.WriteProcessException;
import org.apache.iotdb.db.exception.query.QueryProcessException;
import org.apache.iotdb.db.metadata.idtable.entry.DeviceIDFactory;
import org.apache.iotdb.db.metadata.idtable.entry.IDeviceID;
import org.apache.iotdb.db.metadata.idtable.deviceID.DeviceIDFactory;
import org.apache.iotdb.db.metadata.idtable.deviceID.IDeviceID;
import org.apache.iotdb.db.metadata.utils.ResourceByPathUtils;
import org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertRowNode;
import org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertTabletNode;
Expand Down
2 changes: 1 addition & 1 deletion server/src/main/java/org/apache/iotdb/db/engine/memtable/IMemTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk;
import org.apache.iotdb.db.exception.WriteProcessException;
import org.apache.iotdb.db.exception.query.QueryProcessException;
import org.apache.iotdb.db.metadata.idtable.entry.IDeviceID;
import org.apache.iotdb.db.metadata.idtable.deviceID.IDeviceID;
import org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertRowNode;
import org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertTabletNode;
import org.apache.iotdb.db.qp.physical.crud.InsertRowPlan;
Expand Down
2 changes: 1 addition & 1 deletion server/src/main/java/org/apache/iotdb/db/engine/memtable/PrimitiveMemTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

package org.apache.iotdb.db.engine.memtable;

import org.apache.iotdb.db.metadata.idtable.entry.IDeviceID;
import org.apache.iotdb.db.metadata.idtable.deviceID.IDeviceID;

import java.util.HashMap;
import java.util.Map;
Expand Down
4 changes: 2 additions & 2 deletions server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@
import org.apache.iotdb.db.exception.WriteProcessException;
import org.apache.iotdb.db.exception.WriteProcessRejectException;
import org.apache.iotdb.db.exception.query.QueryProcessException;
import org.apache.iotdb.db.metadata.idtable.entry.DeviceIDFactory;
import org.apache.iotdb.db.metadata.idtable.entry.IDeviceID;
import org.apache.iotdb.db.metadata.idtable.deviceID.DeviceIDFactory;
import org.apache.iotdb.db.metadata.idtable.deviceID.IDeviceID;
import org.apache.iotdb.db.metadata.path.AlignedPath;
import org.apache.iotdb.db.metadata.utils.ResourceByPathUtils;
import org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertRowNode;
Expand Down
16 changes: 12 additions & 4 deletions server/src/main/java/org/apache/iotdb/db/metadata/idtable/AppendOnlyDiskSchemaManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@

import org.apache.iotdb.commons.exception.MetadataException;
import org.apache.iotdb.commons.utils.TestOnly;
import org.apache.iotdb.db.metadata.idtable.entry.DeviceIDFactory;
import org.apache.iotdb.db.metadata.idtable.deviceID.DeviceIDFactory;
import org.apache.iotdb.db.metadata.idtable.entry.DiskSchemaEntry;
import org.apache.iotdb.db.metadata.idtable.entry.SchemaEntry;
import org.apache.iotdb.tsfile.file.metadata.enums.CompressionType;
Expand Down Expand Up @@ -148,7 +148,6 @@ public void recover(IDTable idTable) {
try (FileInputStream inputStream = new FileInputStream(dataFile)) {
// read file version
ReadWriteIOUtils.readString(inputStream);

while (inputStream.available() > 0) {
DiskSchemaEntry cur = DiskSchemaEntry.deserialize(inputStream);
if (!cur.deviceID.equals(DiskSchemaEntry.TOMBSTONE)) {
Expand All @@ -158,7 +157,12 @@ public void recover(IDTable idTable) {
TSEncoding.deserialize(cur.encoding),
CompressionType.deserialize(cur.compressor),
loc);
idTable.putSchemaEntry(cur.deviceID, cur.measurementName, schemaEntry, cur.isAligned);
idTable.putSchemaEntry(
cur.deviceID,
getDevicePathFromSeriesKey(cur.seriesKey, cur.measurementName),
cur.measurementName,
schemaEntry,
cur.isAligned);
}
loc += cur.entrySize;
}
Expand Down Expand Up @@ -248,7 +252,7 @@ private DiskSchemaEntry getDiskSchemaEntryByOffset(long offset) throws IOExcepti
String measurementName = readString();
String deviceID =
DeviceIDFactory.getInstance()
.getDeviceID(seriesKey.substring(0, seriesKey.length() - measurementName.length() - 1))
.getDeviceID(getDevicePathFromSeriesKey(seriesKey, measurementName))
.toStringID();
return new DiskSchemaEntry(
deviceID,
Expand All @@ -267,6 +271,10 @@ private String readString() throws IOException {
return new String(bytes, 0, strLength);
}

private String getDevicePathFromSeriesKey(String seriesKey, String measurement) {
return seriesKey.substring(0, seriesKey.length() - measurement.length() - 1);
}

@Override
public void close() throws IOException {
try {
Expand Down
23 changes: 19 additions & 4 deletions server/src/main/java/org/apache/iotdb/db/metadata/idtable/IDTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,10 @@
import org.apache.iotdb.commons.utils.TestOnly;
import org.apache.iotdb.db.conf.IoTDBConfig;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
import org.apache.iotdb.db.metadata.idtable.deviceID.DeviceIDFactory;
import org.apache.iotdb.db.metadata.idtable.deviceID.IDeviceID;
import org.apache.iotdb.db.metadata.idtable.entry.DeviceEntry;
import org.apache.iotdb.db.metadata.idtable.entry.DeviceIDFactory;
import org.apache.iotdb.db.metadata.idtable.entry.DiskSchemaEntry;
import org.apache.iotdb.db.metadata.idtable.entry.IDeviceID;
import org.apache.iotdb.db.metadata.idtable.entry.SchemaEntry;
import org.apache.iotdb.db.metadata.idtable.entry.TimeseriesID;
import org.apache.iotdb.db.metadata.mnode.IMeasurementMNode;
Expand Down Expand Up @@ -133,6 +133,7 @@ void updateLastCache(
throws MetadataException;

/** clear id table and close file */
@TestOnly
void clear() throws IOException;

/**
Expand All @@ -143,6 +144,14 @@ void updateLastCache(
*/
DeviceEntry getDeviceEntry(String deviceName);

/**
* get device entry from device id
*
* @param deviceID device id of the device path
* @return device entry
*/
DeviceEntry getDeviceEntry(IDeviceID deviceID);

/**
* get schema from device and measurements
*
Expand All @@ -162,13 +171,19 @@ void updateLastCache(
/**
* put schema entry to id table, currently used in recover
*
* @param devicePath device path (can be device id formed path)
* @param deviceID device id
* @param devicePath device path
* @param measurement measurement name
* @param schemaEntry schema entry to put
* @param isAligned is the device aligned
* @throws MetadataException
*/
void putSchemaEntry(
String devicePath, String measurement, SchemaEntry schemaEntry, boolean isAligned)
String deviceID,
String devicePath,
String measurement,
SchemaEntry schemaEntry,
boolean isAligned)
throws MetadataException;

/**
Expand Down
Loading