The UID utility provides various functions to search or modify information in the tsdb-uid table. This includes UID assignments for metrics, tag names and tag values as well as UID meta data, timeseries meta data and tree definitions or data.
Use the UID utility with the command line:
uid <subcommands> [arguments]
Parameters specific to the UID utility include:
The lookup command is the default for uid used to lookup the UID assigned to a name or the name assinged to a UID for a given type.
<kind> <name> <kind> <UID>
./tsdb uid tagk host
tagk host: [0, 0, 1]
The grep sub command performs a regular expression search for the given UID type and returns a list of all UID names that match the expression. Fields required for the grep command include:
| Name | Data Type | Description | Default | Example |
|---|---|---|---|---|
| kind | String | The type of the UID to search for. Must be one of metrics, tagk or tagv
| tagk | |
| expression | String | The regex expression to search with | disk.*write |
grep <kind> '<expression>'
./tsdb uid grep metrics 'disk.*write'
metrics iostat.disk.msec_write: [0, 3, -67] metrics iostat.disk.write_merged: [0, 3, -69] metrics iostat.disk.write_requests: [0, 3, -70] metrics iostat.disk.write_sectors: [0, 3, -68]
This sub command is used to assign IDs to new unique names for metrics, tag names or tag values. Supply a list of one or more values to assign UIDs and the list of assignments will be returned.
| Name | Data Type | Description | Example |
|---|---|---|---|
| kind | String | The type of the UID the names represent. Must be one of metrics, tagk or tagv
| tagk |
| name | String | One or more names to assign UIDs to. Names must not be in quotes and cannot contain spaces. | owner |
assign <kind> <name> [<name>...]
./tsdb uid assign metrics disk.d0 disk.d1 disk.d2 disk.d3
Changes the name of an already assigned UID. If the UID of the given type does not exist, an error will be returned.
Note
After changing a UID name you must flush the cache (see /api/dropcaches) or restart all TSDs for the change to take effect. TSDs do not periodically reload UID maps.
| Name | Data Type | Description | Example |
|---|---|---|---|
| kind | String | The type of the UID the name represent. Must be one of metrics, tagk or tagv
| tagk |
| name | String | The existing UID name | owner |
| newname | String | The new name UID name | server_owner |
rename <kind> <name> <newname>
./tsdb uid rename metrics disk.d0 disk.d0.bytes_read
Removes the mapping of the UID from the tsdb-uid table. Make sure all sources are no longer writing data using the UID and that sufficient time has passed so that users would not query for data that used the UIDs.
Note
After deleting a UID, it may still remain in the caches of running TSD servers. Make sure to flush their caches after deleting an entry.
Warning
Deleting a UID will not delete the underlying data associated with the UIDs (we're working on that). For metrics this is safe, it won't affect queries. But for tag names and values, if a query scans over data containing the old UID, the query will fail with an exception because it can no longer find the name mapping.
| Name | Data Type | Description | Example |
|---|---|---|---|
| kind | String | The type of the UID the name represent. Must be one of metrics, tagk or tagv
| tagk |
| name | String | The existing UID name | owner |
delete <kind> <name>
./tsdb uid delete disk.d0
The UID FSCK command will scan the entire UID table for errors pertaining to name and UID mappings. By default, the run will scan every column in the table and log any errors that were found. With version 2.1 it is possible to fix errors in the table by passing the "fix" flag. UIDMeta objects are skipped during scanning. Possible errors include:
| Error | Description | Fix |
|---|---|---|
| Max ID for metrics is 42 but only 41 entries were found. Maybe 1 IDs were deleted? | This indicates one or more UIDs were not used for mapping entries. If a UID was deleted, this message is normal. If UIDs were not deleted, this can indicate wasted UIDs due to auto-assignments by TSDs where data was coming in too fast. Try assigning UIDs up-front as much as possible. | No fix necessary |
| We found an ID of 42 for metrics but the max ID is only 41! Future IDs may be double-assigned! | If this happens it is usually due to a corruption and indicates the max ID row was not updated properly. | Set the max ID row to the largest detected value |
| Invalid maximum ID for metrics: should be on 8 bytes | Indicates a corruption in the max ID row. | No fix yet. |
| Forward metrics mapping is missing reverse mapping: foo -> 000001 | This may occur if a TSD crashes before the reverse map is written and would only prevent queries from executing against time series using the UID as they would not be able to lookukp the name. | The fix is to restore the missing reverse map. |
| Forward metrics mapping bar -> 000001 is different than reverse mapping: 000001 -> foo | The reverse map points to a different name than the forward map and this should rarely happen. It will be paired with another message. | Depends on the second message |
| Inconsistent forward metrics mapping bar -> 000001 vs bar -> foo / foo -> 000001 | With a forward/reverse miss-match, it is possible that a UID was assigned to multiple names for the same type. If this occurs, then data for two different names has been written to the same time series and that data is effectively corrupt. | The fix is to delete the forward maps for all names that map to the same UID. Then the UID is given a new name that is a dot seperated concatenation of the previous names with an "fsck" prefix. E.g. in the example above we would have a new name of "fsck.bar.foo". This name may be used to access data from the corrupt time series. The next time data is written for the errant names, new UIDs will be assigned to each and new time series created. |
| Duplicate forward metrics mapping bar -> 000002 and null -> foo | In this case the UID was not used more than once but the reverse mapping was incorrect. | The reverse map will be restored, in this case: 000002 -> bar |
| Reverse metrics mapping is missing forward mapping: bar -> 000002 | A reverse map was found without a forward map. The UID may have been deleted. | Remove the reverse map |
| Inconsistent reverse metrics mapping 000003 -> foo vs 000001 -> foo / foo -> 000001 | If an orphaned reverse map points to a resolved forward map, this error occurs. | Remove the reverse map |
Options
fsck [fix] [delete_unknown]
./tsdb uid fsck fix
INFO [main] UidManager: ---------------------------------- INFO [main] UidManager: - Running fsck in FIX mode - INFO [main] UidManager: - Remove Unknowns: false - INFO [main] UidManager: ---------------------------------- INFO [main] UidManager: Maximum ID for metrics: 2 INFO [main] UidManager: Maximum ID for tagk: 4 INFO [main] UidManager: Maximum ID for tagv: 2 ERROR [main] UidManager: Forward tagk mapping is missing reverse mapping: bar -> 000004 INFO [main] UidManager: FIX: Restoring tagk reverse mapping: 000004 -> bar ERROR [main] UidManager: Inconsistent reverse tagk mapping 000003 -> bar vs 000004 -> bar / bar -> 000004 INFO [main] UidManager: FIX: Removed tagk reverse mapping: 000003 -> bar ERROR [main] UidManager: tagk: Found 2 errors. INFO [main] UidManager: 17 KVs analyzed in 334ms (~50 KV/s) WARN [main] UidManager: 2 errors found.
This command will run through the entire data table, scanning each row of timeseries data and generate missing TSMeta objects and UIDMeta objects or update the created timestamps for each object type if necessary. Use this command after enabling meta tracking with existing data or if you suspect that some timeseries may not have been indexed properly. The command will also push new or updated meta entries to a search engine if a plugin has been configured. If existing meta is corrupted, meaning the TSD is unable to deserialize the object, it will be replaced with a new entry.
It is safe to run this command at any time as it will not destroy or overwrite valid data. (Unless you modify columns directly in HBase in a manner inconsistent with the meta data formats). The utility will split the data table into chunks processed by multiple threads so the more cores in your processor, the faster the command will complete.
metasync
./tsdb uid metasync
This sub command will mark all TSMeta and UIDMeta objects for deletion in the UID table. This is useful for downgrading from 2.0 to a 1.x version or simply flushing all meta data and starting over with a metasync.
metapurge
./tsdb uid metapurge
Runs through the list of TSMeta objects in the UID table and processes each through all configured and enabled trees to compile branches. This command may be run at any time and will not affect existing objects.
treesync
./tsdb uid treesync
Removes all branches, collision, not matched data and optionally the tree definition itself for a given tree. Parameters include:
| Name | Data Type | Description | Example |
|---|---|---|---|
| id | Integer | ID of the tree to purge | 1 |
| definition | Flag | Add this literal after the ID to delete the definition of the tree as well as the data | definition |
treepurge <id> [definition]
./tsdb uid treepurge 1
© 2010–2016 The OpenTSDB Authors
Licensed under the GNU LGPLv2.1+ and GPLv3+ licenses.
http://opentsdb.net/docs/build/html/user_guide/cli/uid.html